Information processing device, information processing method, and computer program

ABSTRACT

An information processing device according to the present technology includes an action recognition unit that recognizes an operation action of a user based on sensor information, and an action representation generation unit that analyzes operation action data showing the operation action of the user recognized by the action recognition unit to generate an action segment represented by a meaning and content of the operation action from the operation action data.

TECHNICAL FIELD

The present disclosure relates to an information processing device thatprocesses a user's action records, an information processing method, anda computer program.

BACKGROUND ART

A technology to recognize a user's operation action from sensorinformation acquired by using various sensing technologies is proposed.The recognized user's operation action is automatically recorded as anaction log and can be represented by various techniques, for example,reproducing the operation action by animation such as an avatar, showinga user's movement locus on a map, or using an index abstracting variousoperation actions for representation.

CITATION LIST Patent Literature

-   Patent Literature 1: JP 2008-3655A

SUMMARY OF INVENTION Technical Problem

However, when an action log is reproduced by animation such as an avatarusing an action recording device like, for example, a motion capture, avery large-scale device will be needed. On the other hand, an action loggenerated by using a small sensor containing recording device such as asmartphone limits the types of action that can be recorded/recognizedand thus, it is difficult to present an action record that is valuableto the user. Therefore, an action log is generally shown as a user'smovement locus on a map or displayed as an action record converted tothe amount of activity like a health index.

Therefore, a proposal of the representation technique to present anaction log recorded by a small sensor containing recording device to theuser in a manner that is easy to understand has been sought.

Solution to Problem

According to the present disclosure, there is provided an informationprocessing device including an action recognition unit that recognizesan operation action of a user based on sensor information, and an actionrepresentation generation unit that analyzes operation action datashowing the operation action of the user recognized by the actionrecognition unit to generate an action segment represented by a meaningand content of the operation action from the operation action data.

According to the present disclosure, there is provided an informationprocessing device including an action recognition unit that recognizesan operation action of a user based on sensor information, an actionrepresentation generation unit that generates an action segmentconstituting an action log from operation action data showing theoperation action of the user recognized by the action recognition unitbased on operation action estimation information that decides theoperation action, and a feedback adjustment unit that corrects theoperation action estimation information based on correction feedbackfrom the user to the action segment generated by the actionrepresentation generation unit.

According to the present disclosure, there is provided an informationprocessing method including a step for recognizing an operation actionof a user based on sensor information, and a step for analyzingoperation action data showing the recognized operation action of theuser to generate an action segment represented by a meaning and contentof the operation action from the operation action data.

According to the present disclosure, there is provided an informationprocessing method including a step for recognizing an operation actionof a user based on sensor information, a step for generating an actionsegment constituting an action log from operation action data showingthe recognized operation action of the user based on operation actionestimation information that decides the operation action, and a step forcorrecting the operation action estimation information based oncorrection feedback from the user to the action segment.

According to the present disclosure, there is provided a computerprogram for causing a computer to function as an information processingdevice including an action recognition unit that recognizes an operationaction of a user based on sensor information, and an actionrepresentation generation unit that analyzes operation action datashowing the operation action of the user recognized by the actionrecognition unit to generate an action segment represented by a meaningand content of the operation action from the operation action data.

According to the present disclosure, there is provided a computerprogram for causing a computer to function as an information processingdevice including an action recognition unit that recognizes an operationaction of a user based on sensor information, an action representationgeneration unit that generates an action segment constituting an actionlog from operation action data showing the operation action of the userrecognized by the action recognition unit based on operation actionestimation information that decides the operation action, and a feedbackadjustment unit that corrects the operation action estimationinformation based on correction feedback from the user to the actionsegment generated by the action representation generation unit.

According to the present disclosure, operation action data showing auser's operation action recognized by an action recognition unit basedon sensor information is analyzed by an action representation generationunit to generate an action segment represented by the meaning andcontent of the operation action from the operation action data. Bydisplaying an action log with the action segment represented by themeaning and content of the operation action, information can bepresented to the user in a manner that is easy to understand.

Advantageous Effects of Invention

According to the present disclosure, as described above, a recordedaction log can be presented to the user in a manner that is easy tounderstand.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an explanatory view showing the configuration of an action logdisplay system according to an embodiment of the present disclosure.

FIG. 2 is a functional block diagram showing a functional configurationof the action log display system according to the embodiment.

FIG. 3 is an explanatory view showing an example of a context leveldictionary.

FIG. 4 is an explanatory view showing a case when an action segment isgenerated from operation action data by contextual analysis and thedisplay of the action log is changed by changing a segmentation grainsize of the action segment.

FIG. 5 is an explanatory view showing another example of the case whenthe action segment is generated from operation action data by thecontextual analysis and the display of the action log is changed bychanging the segmentation grain size of the action segment.

FIG. 6 is an explanatory view showing a case when an action segment isgenerated from operation action data by combining the contextualanalysis and a time width and the display of the action log is changedby changing the segmentation grain size of the action segment.

FIG. 7 is an explanatory view showing a case when an action segment isgenerated from operation action data by combining the contextualanalysis, the time width, and position changes and the display of theaction log is changed by changing the segmentation grain size of theaction segment.

FIG. 8 is a flow chart showing the flow of overall processing of actionrecognition.

FIG. 9 is a flow chart showing processing by a living action recognitionunit.

FIG. 10 is a flow chart showing processing by a hierarchical structurejudgment unit.

FIG. 11 is a flow chart showing action segment generation processing.

FIG. 12 is a flow chart showing the action segment generationprocessing.

FIG. 13 is an explanatory view illustrating a method of attachinghierarchical information to the action segment.

FIG. 14 is functional block diagram showing the functional configurationof an analysis server.

FIG. 15 is an explanatory view showing an example of the representationof the action log.

FIG. 16 is an explanatory view showing a display example of the actionlog when an action log display application is activated.

FIG. 17 is an explanatory view showing a display example of a calendar.

FIG. 18 is an explanatory view showing a display example when a map iscaused to display position information corresponding to the action logby operating a map button.

FIG. 19 is an explanatory view showing a state in which a correctionscreen to correct the action segment to be corrected is displayed.

FIG. 20 is an explanatory view showing an example of the correctionscreen to correct an operation action.

FIG. 21 is an explanatory view showing an example of a method ofcombining action segments.

FIG. 22 is an explanatory view showing an example of another method ofdeciding operation content of the action segment after combination.

FIG. 23 is an explanatory view showing an example of a division methodby time settings of the action segment.

FIG. 24 is an explanatory view showing an example of the division methodbased on hierarchical information of the action segment.

FIG. 25 is an explanatory view showing a case when display roughness ischanged by using a slider.

FIG. 26 is an explanatory view showing a case when the display roughnessis changed by using a zoom button.

FIG. 27 is an explanatory view showing a display change of the actionsegment when a button of “work details” is checked in an action typeselection list.

FIG. 28 is an explanatory view showing a display change of the actionsegment when a button of “shopping details” is checked in the actiontype selection list.

FIG. 29 is an explanatory view showing a display change of the actionsegment when a button of “movement details” is checked in the actiontype selection list.

FIG. 30 is an explanatory view showing a display change of the actionsegment when a button of “uniform details” is checked in the action typeselection list.

FIG. 31 is an explanatory view showing a positional configurationexample of a display grain size setting unit provided with a slider thatsets the display roughness of the action segment for each type ofaction.

FIG. 32 is an explanatory view showing a method of deleting an actionsegment from the action log.

FIG. 33 is an explanatory view showing a method of posting content ofthe action segment of the action log to a posting site.

FIG. 34 is an explanatory view showing a positional configurationexample of a setting screen to make various settings about the actionlog display application.

FIG. 35 is a flow chart showing an example of action recognitionprocessing by the living action recognition unit.

FIG. 36 is an explanatory view showing operation action estimationinformation showing a relationship between a weighting factor dependingon the location and a probability distribution of each action.

FIG. 37 is an explanatory view providing an overview of reflectionprocessing of correction feedback.

FIG. 38 is a flow chart showing the reflection processing of correctionfeedback of an action.

FIG. 39 is an explanatory view illustrating corrections of the operationaction estimation information based on the processing in FIG. 38.

FIG. 40 is a flow chart showing other reflection processing ofcorrection feedback of the action.

FIG. 41 is a flow chart showing the reflection processing of correctionfeedback of the action and position information.

FIG. 42 is an explanatory view illustrating personal modeling of anaction pattern by a typical action pattern generation unit.

FIG. 43 is an explanatory view illustrating a position display techniqueby determining a medium/means of transport.

FIG. 44 is an explanatory view illustrating line estimation processing.

FIG. 45 is an explanatory view illustrating station name selectionprocessing.

FIG. 46 is a block diagram showing a hardware configuration example ofan action recording device according to the embodiment.

DESCRIPTION OF EMBODIMENT

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the drawings, elements that have substantiallythe same function and structure are denoted with the same referencesigns, and repeated explanation is omitted.

The description will be provided in the order shown below:

<1. Overview of Action Log Display System>

<2. Functional Configuration of Action Log Display System>

[2-1. Action Recording Device]

[2-2. Action Log Server]

[2-3. Analysis Server]

<3. Action Segment Generation Processing>

[3-1. Relationship between Operation Action and Meaning/Content thereof]

[3-2. Action Segment Generation Processing]

(Example 1: Generation of an action segment by the contextual analysis)

(Example 2: Generation of an action segment by combining the contextualanalysis and the time width)

(Example 3: Generation of an action segment by combining the contextualanalysis, the time width, and position changes)

[3-3. Action Recognition Processing]

[3-4. Processing Content of Action Representation Generation Unit]

<4. Action Recording and Display Application>

[4-1. Representation of Action Log Based on Action Segment]

[4-2. Browsing Action]

[4-3. Correcting Action]

[4-4. Combining Actions]

[4-5. Dividing Action]

[4-6. Representation of Action Segment Based on Segmentation Grain Size]

[4-7. Deleting Action]

[4-8. Posting Action]

[4-9. Action Log Acquisition Stop Processing]

[4-10. Updating Display Content]

<5. Reflection Processing of Correction Feedback>

[5-1. Properties of Correction Feedback]

[5-2. Action Recognition Processing]

[5-3. Reflection Processing of Correction Feedback]

(5-3-1. Overview of reflection processing of correction feedback)

(5-3-2. Reflection processing of correction feedback of an action)

(5-3-3. Reflection processing of correction feedback of an action andposition information)

<6. Others>

[6-1. Personal Modeling of Action Pattern]

[6-2 Position Display Technique by Moving Medium/Means Determination]

(6-2-1. Line estimation processing)

(6-2-2. Station name selection processing)

<7. Exemplary Hardware Configuration>

<1. Overview of Action Log Display System>

First, an overview of an action log display system according to anembodiment of the present disclosure will be provided with reference toFIG. 1. FIG. 1 is an explanatory view showing an outline configurationof the action log display system according to the present embodiment.

The action log display system according to the present embodimentrealizes a representation technique that presents an action log recordedby a recording device 100 containing a small sensor (hereinafter,referred to as an “action recording device”) to the user in a mannerthat is easy to understand. As the action recording device 100, forexample, a mobile terminal such as a mobile phone, PDA (Personal DigitalAssistant), and smartphone can be used. The action recording device 100is provided with at least one sensor to sense conditions or an action ofa user holding the device. The action recording device 100 estimates anoperation action of the user based on sensor information acquired by thesensor and transmits the operation action to an action log server 200 asan action log. In this manner, the action log of the user is accumulatedin the action log server 200.

An action log analyzed by the action recording device 100 and stored inthe action log server 200 records an operation like, for example,“meal”, “movement”, and “sleep” together with the action time, positioninformation and the like. An action log display system according to thepresent embodiment further analyzes an action log representing theoperation content by an analysis server 300 to recognize the meaning ofaction and generates information (action segment) to which the meaningof action is added. The action segment is unit information as aneasy-to-understand representation for the user of an action log. Insteadof simply presenting an action log to the user, the action segment canpresent an action log in a manner that conveys the meaning of action.

An action log analyzed by the analysis server 300 and presented to theuser can be corrected by the user. In addition, data of the presentedaction log can be combined, divided, or deleted by generating an actionsegment. The presented action log can also be posted to a posting site.Thus, by using an action log display system according to the presentembodiment, an action log acquired as an operation can be analyzed andpresented to the user in an easy-to-understand manner. The configurationand function of an action log display system according to the presentembodiment will be described in detail below.

<2. Functional Configuration of Action Log Display System>

FIG. 2 shows the functional configuration of the action log displaysystem according to the present embodiment. The action log displaysystem includes, as described above, the action recording device 100that records a user's operation action, the action log server 200 thatmanages an action log recorded by the action recording device 100, andthe analysis server 300 that analyzes an action log to generate anaction segment.

[2-1. Action Recording Device]

The action recording device 100 includes sensors 110, an actionrecognition unit 120, a client interface unit 130, an actionrepresentation processing unit 140, a display unit 150, and an inputunit 160.

The sensors 110 are devices that sense a user's action or conditions andare installed in the action recording device 100. As the sensors 110,for example, an acceleration sensor, gyro sensor, magnetic field sensor,atmospheric pressure sensor, illuminance sensor, temperature sensor,microphone and the like can be used. As the sensors 110, alatitude/longitude acquisition sensor that acquires thelatitude/longitude can also be installed. As the latitude/longitudeacquisition sensor, for example, not only GPS (Global PositioningSystem) or WiFi, but also base station information of othercommunication networks or information such as RFID and images may beused. The sensors 110 output detected information to the actionrecognition unit 120 as sensor information.

The action recognition unit 120 estimates a user's action based onsensor information. The action recognition unit 120 includes a sensorcontroller 122 and an operation action recognition unit 124. The sensorcontroller 122 controls the sensor 110, the CPU or the overall system tocause sensing by the sensor 110 to operate effectively. The sensorcontroller 122 controls the above devices based on recognition resultsby the sensor 110 or the operation action recognition unit 124.

The operation action recognition unit 124 recognizes a user's action orconditions by performing signal processing or statistical processing ofsensor information. The action recording device 100 holds acorrespondence between an action model as information about a user'saction obtained as a result of processing sensor information and anoperation action in advance. When action parameters are obtained byprocessing sensor information, the operation action recognition unit 124identifies an operation action corresponding to the parameters. Then,the operation action recognition unit 124 associates the identifiedoperation action and the action time period, action time, positioninformation and the like and outputs the associated information to theclient interface unit 130 as operation action data. The operation actiondata is uploaded from the client interface unit 130 to the action logserver 200.

The client interface unit 130 transmits/receives information between theaction recording device 100, and the action log server 200 and theanalysis server 300. For example, the client interface unit 130transmits operation action data input from the action recognition unit120 to the action log server 200 or outputs an analysis result receivedfrom the analysis server 300 to the action representation processingunit 140. Also, the client interface unit 130 transmits feedbackinformation from the user input through the input unit 160 to theanalysis server 300.

The action representation processing unit 140 is a functional unit thatdisplays an action log or processes feedback information from the userand includes a display processing unit 142 and an input informationprocessing unit 144. The display processing unit 142 performs processingto display an analysis result by the analysis server 300 input from theclient interface unit 130 in the display unit 150. The input informationprocessing unit 144 performs processing to transmit feedback informationfrom the user for an action log input from the input unit 160 to theanalysis server 300 via the client interface unit 130.

The display unit 150 is an output device that displays information andcan be configured by, for example, a liquid crystal display, organic ELdisplay or the like. For example, an action log processed for display bythe display processing unit 142 is displayed in the display unit 150.

The input unit 160 is an input device to input information and, forexample, a touch panel, keyboard, hardware button or the like can beused. In the present embodiment, it is assumed that the display surfaceof the display unit 150 is provided with a touch panel as the input unit160. In this case, the user can input information by, for example,bringing an operation body such as a finger or touch pen into contactwith the display surface of the display unit 150 or moving the operationbody brought into contact with the display surface. Information inputfrom the input unit 160 is output to the input information processingunit 144.

[2-2. Action Log Server]

The action log server 200 includes a log server interface unit 210 andan action log DB 220.

The log server interface unit 210 transmits/receives information betweenthe action log server 200, and the action recording device 100 and theanalysis server 300. For example, the log server interface unit 210records operation action data received from the action recording device100 in the action log DB 220 or acquires operation action data inaccordance with a transmission request from the analysis server 300 fromthe action log DB 220 and transmits the operation action data to theanalysis server 300.

The action log DB 220 is a storage unit that stores operation actiondata of the user acquired by the action recording device 100. Inoperation action data stored in the action log DB 220, as describedabove, the operation action identified by the operation actionrecognition unit 124 and the action time period, action time, positioninformation and the like are associated and stored in the action log DB220 in, for example, chronological order.

[2-3. Analysis Server]

The analysis server 300 includes an analysis server interface unit 310,an action representation generation unit 320, and a data management unit330.

The analysis server interface unit 310 transmits/receives informationbetween the analysis server 300, and the action recording device 100 andthe action log server 200. For example, the analysis server interfaceunit 310 receives an analysis instruction (analysis request) of anaction log from the action recording device 100 or transmits atransmission request of necessary operation action data in accordancewith an analysis request The analysis server interface unit 310 alsoreceives feedback information from the user of an action log from theaction recording device 100.

The action representation generation unit 320 analyzes operation actiondata to understand the meaning thereof and generates an action segmentto which the meaning and content is added. The action representationgeneration unit 320 includes a living action recognition unit 321 and ahierarchical structure judgment unit 322. The living action recognitionunit 321 generates an action segment from an action log includingoperation action data. The living action recognition unit 321 analysesthe meaning and content of operation action data arranged inchronological order based on relationships between data and the timeperiod, time and the like of data. Then, the living action recognitionunit 321 selects data classified as the most detailed meaning andcontent of analyzed meaning and content as the action segment. Thegenerated action segment is output to the data management unit 330 andheld there.

The hierarchical structure judgment unit 322 judges a hierarchicalstructure about the meaning and content of an action segment generatedby the living action recognition unit 321 and attaches hierarchicalinformation representing a hierarchical relationship of the meaning andcontent to the action segment. Hierarchical information is hierarchicalmeaning information attached to an action segment by processingdescribed later. Hierarchical information may be, for example,information using a normalized value as a key or information using IDidentifying the level of meaning information as a direct key. An actionsegment to which hierarchical information is attached is also called ahierarchical information attached action segment. Hierarchicalrelationships of the meaning and content of action are stored in thedata management unit 330. The hierarchical structure judgment unit 322outputs a hierarchical information attached action segment to the datamanagement unit 330 via the living action recognition unit 321. Thefunction of the action representation generation unit 320 and details ofprocessing content thereby will be described later.

The data management unit 330 manages an action segment generated by theaction representation generation unit 320. The data management unit 330includes a data acquisition unit 331, a feedback adjustment unit 332, ananalysis parameter DB 333, a unit data storage DB 334, and ahierarchical information attached data storage DB 335.

The data acquisition unit 331 transmits/receives data to/from the actionrepresentation generation unit 320. The data acquisition unit 331records an action segment transmitted from the action representationgeneration unit 320 in the unit data storage DB 334 or records ahierarchical information attached action segment in the hierarchicalinformation attached data storage DB 335. The data acquisition unit 331acquires the specified action segment in accordance with a request fromthe action representation generation unit 320 from the unit data storageDB 334 or the hierarchical information attached data storage DB 335 andoutputs the action segment to the action representation generation unit320.

The feedback adjustment unit 332 reflects feedback information receivedfrom the action recording device 100 in analysis parameters used foranalyzing the meaning and content of operation action data. The feedbackinformation represents content of processing such as corrections made bythe user on an action log displayed in the display unit 150 of theaction recording device 100. The feedback adjustment unit 332 correctsanalysis parameters using feedback information so that the meaning andcontent of a user's action can be recognized more correctly.

The analysis parameter DB 333 is a storage unit that holds analysisparameters used for analyzing the meaning and content of operationaction data. In the analysis parameter DB 333, for example, acorrespondence between an operation action and the meaning and contentis stored as analysis parameters. Information stored in the analysisparameter DB 333 can be referenced by both of the living actionrecognition unit 321 and the hierarchical structure judgment unit 322.Analysis parameters are updated when necessary based on feedbackinformation from the user.

The unit data storage DB 334 stores an action segment generated by theaction representation generation unit 320. The action segment stored inthe unit data storage DB 334 is a segment (unit segment) of the minimumunit necessary for recognition.

The hierarchical information attached data storage DB 335 stores anaction segment in which hierarchical information is attached to anaction segment generated by the action representation generation unit320. The action segment stored in the hierarchical information attacheddata storage DB 335 is a hierarchical information attached actionsegment to which hierarchical information representing a hierarchicalstructure of the meaning and content of action is attached by thehierarchical structure judgment unit 322. The recording timing of anaction segment to which hierarchical information is attached may be, forexample, when requested by an application or analysis results of aplurality of segmentation grain sizes may be recorded in advance by theaction representation generation unit 320.

That is, the action representation generation unit 320 and the datamanagement unit 330 function as information processing devices thatanalyze the meaning and content of operation action data generated bythe action recording device 100 to present information that is easy forthe user to understand.

<3. Action Segment Generation Processing>

In an action log display system according to the present embodiment, themeaning and content of an operation action generated by the actionrecording device 100 is analyzed by the analysis server 300 to generatean action segment based on the meaning and content of action.Hierarchical information about the meaning and content of action canalso be attached to an action segment and the display form of an actionlog can also be changed easily based on the hierarchical information.First, generation processing of an action segment will be describedbased on FIGS. 3 to 13.

[3-1. Relationship Between Operation Action and Meaning/Content Thereof]

The action recording device 100 analyzes an operation action, forexample, “meal”, “movement”, or “sleep”. The analysis server 300analyzes content of each operation action more deeply using operationaction data containing the operation action. The analysis of the meaningand content of the operation action is conducted by using, for example,as shown in FIG. 3, a context level dictionary.

If, among the operation actions of “meal”, “movement”, and “sleep”, theoperation action of “movement” is taken up, As shown in FIG. 3, themeaning and content thereof changes depending on what kind of movement.For example, when “moving on foot”, the user can take action ofcontinuing to “walk” or then change to an action of “halting”. When, forexample, “waiting for means of transport” on the move, action ofcontinuing to “wait for means of transport” can be taken. Further, when,for example, “moving by train”, the movement can be made an action of“movement by train on a single line”. Further, a state in which a stateof “movement by rain” continues or a state of “train stopped” in whichthe train on which the user moves stops can be assumed. Alternatively,action of “trains changed” after “movement by train” can be taken or astate transition to “waiting for means of transport” can also takeplace.

Then, a further action of “walking” can be associated with a “walking”action or “changing trains” action and a further action of “stopping”can be associated with a “halting” action, a “waiting for means oftransport” state, or a “train stop” state. A “train” as a means oftransport can further be associated with a state of “moving by train”.

Thus, action meta information at an operation action level could changeto, as shown in FIG. 3, a higher level of action meta informationdepending on the context. An action log display system in the presentembodiment can analyze the meaning and content of an action based on therelationship between an operation action and the meaning and content andso can present an action log that is easy for the user to understand. Inaddition, by attaching the hierarchical relationship as hierarchicalinformation to an action segment regarding the meaning and content ofthe operation action, the segmentation grain size of an action logdescribed later can easily be changed.

In the present embodiment, the ontology/semantic technology is appliedto the recognition of the meaning and content of an operation action tosegment the action recognition that judges “context” in an “action” andoperation action data. Ontology systematically represents the concept ofrelations between words and in the present embodiment, for example, asshown in FIG. 3, the concept of relations between actions issystematized. Then, using the systematized concept of relations betweenactions, the meaning of an action or the relationship of actions isunderstood by applying the semantic technology and recognitionprocessing of the meaning and content of the operation action isperformed based on the understanding. For example, by judging thecontext using properties that a transition from some action (forexample, a meal) to some action (for example, work) is likely to occuraccompanying, for example, an operation action of “movement on foot”, anaction segment fitting to the feeling of the user can be generated.

[3-2. Action Segment Generation Processing]

Example 1 Generation of an Action Segment by the Contextual Analysis

As a concrete example of generation processing of an action segmentusing the relationship between the operation action and the meaning andcontent, generation processing of an action segment by the contextualanalysis will be described based on FIGS. 4 and 5. FIG. 4 is anexplanatory view showing a case when an action segment is generated fromoperation action data by the contextual analysis and the display of theaction log is changed by changing a segmentation grain size of theaction segment. FIG. 5 is an explanatory view showing another example ofthe case when the action segment is generated from operation action databy the contextual analysis and the display of the action log is changedby changing the segmentation grain size of the action segment.

As shown in FIG. 4, it is assumed that an action log including operationaction data is acquired by the operation action recognition unit 124.The operation action data is arranged from left to right inchronological order. The operation action data is an operation actionrecognized based on sensor information of the sensors 110 and an actionlog is represented by a simple operation action like “walked”,“stopped”, and “got on a train”. The living action recognition unit 321recognizes the meaning of each piece of operation action data or therelationship between operation action data using a dictionary as shownin FIG. 3 from such an action log.

For example, a state of short “stopped” of a predetermined time orshorter between operation action data of “got on a train” is estimatedto be a “train stopped (at a station)” state. A state of short “walked”of a predetermined time or shorter between operation action data of “goton a train” is estimated to be a “changing trains” action. Further,“stopped” of operation action data immediately before operation actiondata of “got on a train” is estimated to be a state of “waited for atrain”.

By using the action time of operation action data, the action can beestimated more appropriately. Regarding an action of “movement bytrain”, for example, the meaning of action of “going to office” or“going to school” can be estimated if the action time is a morning hour(for example, from 6 am to 10 am) and the meaning of action of “goinghome” can be estimated if the action time is an evening hour (forexample, from 5 pm to 8 pm). Similarly, regarding an action of “meal”,the meaning of action of “breakfast” can be estimated if the action timeis a morning hour, “lunch” can be estimated if the action time is anhour around noon, “supper” can be estimated if the action time is anevening hour.

Thus, an action log including action segments as shown in FIG. 4 isgenerated by an action log including operation action data beinganalyzed by the living action recognition unit 321. An action segment isdata representing an operation action to which operation content isadded and is a chunk of consecutive operation actions having the samemeaning and content. An action segment generated to match operationaction data includes unit segments to which detailed operation contentis added. Therefore, if an action log is represented by action segments,as shown in FIG. 4, the moving state while riding on a train can also beknown.

Then, by acquiring hierarchical action meta information at an operationaction level from the context level dictionary shown in FIG. 3 tohierarchically change the segmentation grain size as a parameter todetermine the roughness of segmentation of the action segment, thedisplay of the action segment can be changed. With an increasingsegmentation grain size, a plurality of action segments that can beconsidered to be one action is combined to produce an action segmentrepresenting rough operation content. On the other hand, with adecreasing segmentation grain size, an action segment approaches theunit segment.

For example, as shown in FIG. 4, the segmentation grain size of anaction log represented by unit segments is increased. In a state of agrain size 1-1 reached by increasing the segmentation grain size fromthe segmentation grain size of the unit segment, a series of operationsof “got on a train”, “train stopped”, and “got on a train” arerepresented by one action segment of “got on a train on some line”. In astate of a grain size 1-2 reached by further increasing the segmentationgrain size, a series of operations of “got on a train on some line”,“trains changed”, and “got on a train on some line” are represented byone action segment of “movement by train”. In a state of a grain size1-3 reached by further increasing the segmentation grain size, a seriesof operations of “walked”, “waited for a train”, “movement by train” and“walked” are represented by one action segment of “moved”.

By changing the segmentation grain size based on hierarchical actionmeta information at the operation action level in this manner, theaction log can be displayed at an operation action level that is easyfor the user to view.

An action segment concerning movement is described with reference toFIG. 4, but an action log can also be displayed similarly for otheractions. Assume that, for example, as shown in FIG. 5, “action model X”representing a conspicuously unsteady motion appearing in a shoppingaction or the like and an operation action of “had a meal” arerecognized by the operation action recognition unit 124 as operationaction data. “Action model X” normally means the class of shopping, butthe meaning thereof changes in accordance with a prior or subsequentdetermination result.

In the example shown in FIG. 5, “action model X” is present before andafter “had a meal”. In this case, the living action recognition unit 321recognizes the action model X as “accepted” and “paid the bill” asoperations performed before and after an operation of “have a meal” fromoperation action data before or after the action model X. Thus, anaction log that is easy for the user to understand can be presented bythe meaning and content being added to the user's unsteady motion fromthe prior or subsequent operation. Also in this case, a series ofactions of “accepted”, “had a meal”, and “paid the bill” can berepresented as one action segment of “had a meal” by increasing thesegmentation grain size.

Example 2 Generation of an Action Segment by Combining the ContextualAnalysis and the Time Width

Next, a concrete example when an action segment is generated byconsidering, in addition to the contextual analysis, the time width willbe described based on FIG. 6. FIG. 6 is an explanatory view showing acase when an action segment is generated from operation action data bycombining the contextual analysis and the time width and the display ofthe action log is changed by changing the segmentation grain size of theaction segment.

As shown in FIG. 6, it is assumed that an action log including operationaction data is acquired by the operation action recognition unit 124. Inthe present example, an action log is represented by simple operationactions like “desk work”, “walked”, “advance arrangements”, and “meal”as operation action data. The living action recognition unit 321recognizes the meaning of each piece of operation action data or therelationship between operation action data using a dictionary as shownin FIG. 3 and also recognizes the meaning and content thereof byconsidering the time width of the operation action data.

FIG. 6 shows an example of the action log in a company. What kind ofwalking the operation action data of “walked” is about can be recognizedfrom prior or subsequent operation action data, but can also beestimated based on a walking time. An operation action of short “walked”in a company normally does not have any special meaning. However, if thewalking continues for a predetermined time or longer, the walking isestimated not to be simple movement on the floor, but to be movementbetween premises. Thus, action segments generated by combining thecontextual analysis using a dictionary and the time width of operationaction data are as shown in FIG. 6. The operation action data of“walked” is divided into “short walking” and “movement between premises”depending on the time width thereof.

When action segments are generated, like in the above case, the displayof the action log including the action segments can easily be changed bychanging the segmentation grain size. For example, in a state of a grainsize 2-1 reached by increasing the segmentation grain size from thesegmentation grain size of the unit segment, a series of operations of“desk work”, “short walking”, “advance arrangements”, “short walking”,and “desk work” are represented as one action segment of “worked”. Inthis case, “short walking” is combined into one action segment of“worked” and thus, the action segment of “movement between premises” maybe displayed simply as “movement”.

In a state of a grain size 2-2 reached by further increasing thesegmentation grain size, a series of operations of “worked”, “meal”,“worked”, “movement”, and “worked” are represented as one action segmentof “was in company”. By changing the segmentation grain size based onhierarchical action meta information at the operation action level inthis manner, the action log can be displayed at an operation actionlevel that is easy for the user to view.

Example 3 Generation of an Action Segment by Combining the ContextualAnalysis, the Time Width, and Position Changes

Next, a concrete example when an action segment is generated byconsidering, in addition to the contextual analysis and time width,position changes will be described based on FIG. 7. FIG. 7 is anexplanatory view showing a case when an action segment is generated fromoperation action data by combining the contextual analysis, the timewidth, and position changes and the display of the action log is changedby changing the segmentation grain size of the action segment.

As shown in FIG. 7, it is assumed that an action log including operationaction data is acquired by the operation action recognition unit 124. Inthe present example, an action log is represented by simple operationactions like “did shopping”, and “walked” as operation action data. Theliving action recognition unit 321 recognizes the meaning of each pieceof operation action data or the relationship between operation actiondata using a dictionary as shown in FIG. 3 and also recognizes themeaning and content thereof by considering the time width of theoperation action data and position changes of the action recordingdevice 100 (that is, the user).

FIG. 7 shows an example of the action log in shopping. What kind ofwalking the operation action data of “walked” is about can be recognizedfrom prior or subsequent operation action data, but can also beestimated in detail based on a walking time and position changes.

If, for example, the operation actions before and after the operationaction data of “walked” are “did shopping” and a movement time t is t1(for example, 35 s) or more and position changes of the action recordingdevice 100 carried by the user are measured, the user is estimated to“move between shops”. Also, if, for example, the operation actionsbefore and after the operation action data of “walked” are “didshopping” and the movement time t is t2 (for example, 20 s) or more andshorter than t1, and no position change of the action recording device100 is measured, the user is estimated to “move between floors” duringshopping. Further, if, for example, the operation actions before andafter the operation action data of “walked” are “did shopping” and themovement time t is t3 (for example, 5 s) or more and shorter than t2,and no position change of the action recording device 100 is measured,the user is estimated to “move in a shop” during shopping.

Thus, if the meaning and content of operation action data is recognizedby combining the contextual analysis, time width, and position changes,as shown in FIG. 7, action segments including three action segments of“movement in a shop (SG1)”, “movement on floors (SG2)”, and “movementbetween shops (SG3)” are generated from operation action data of“walked”.

When action segments are generated, like in the above case, the displayof the action log including the action segments can easily be changed bychanging the segmentation grain size. For example, in a state of a grainsize 3-1 reached by increasing the segmentation grain size from thesegmentation grain size of the unit segment, among action segmentsconcerning walking, the action segment SG1 with the shortest walkingtime is combined with the action segments of “did shopping” precedingand succeeding the action segment SG1. These action segments arerepresented by an action segment as a series of operations of “didshopping”. In this case, “movement in a shop” is combined into oneaction segment of “did shopping”, other action segments concerningwalking may be displayed simply as “movement”.

In a state of a grain size 3-2 reached by further increasing thesegmentation grain size, among action segments concerning walking, theaction segment SG2 with the shortest walking time next to the actionsegment SG1 is combined with the action segments of “did shopping”preceding and succeeding the action segment SG2. Then, in a state of agrain size 3-3 reached by further increasing the segmentation grainsize, among action segments concerning walking, the action segment SG3with the longest walking time is also combined with the action segmentsof “did shopping” preceding and succeeding the action segment SG3.Accordingly, a series of operations of “did shopping” and “walked” arerepresented as one action segment. By changing the segmentation grainsize based on hierarchical action meta information at the operationaction level in this manner, the action log can be displayed at anoperation action level that is easy for the user to view.

[3-3. Action Recognition Processing]

Processing to generate an action segment from operation action data willbe described in detail based on FIGS. 8 to 10. FIG. 8 is a flow chartshowing the flow of overall processing of action recognition. FIG. 9 isa flow chart showing processing by the living action recognition unit321. FIG. 10 is a flow chart showing processing by the hierarchicalstructure judgment unit 322.

Action recognition processing includes, as shown in FIG. 8, operationaction data creation processing (S100, S110) performed by the actionrecognition unit 120 of the action recording device 100 and actionsegment generation processing (S120 to S140) performed by the actionrepresentation generation unit 320 and the data management unit 330 ofthe analysis server 300.

The operation action recognition unit 124 of the action recognition unit120 having acquired sensor information from the sensors 110 startscreation processing of operation action data (S100). The operationaction data creation processing can be performed by using an existingtechnique. After creating operation action data, the operation actionrecognition unit 124 outputs the operation action data to the action logserver 200 (S110). In this manner, operation action data constituting anaction log of the user is accumulated in the action log server 200.Incidentally, the action recognition unit 120 may generate, as operationaction data, not only action information at the operation level, butalso information including, for example, time information, locationinformation, an operation history of devices and the like.

When operation action data is created, the action representationgeneration unit 320 of the analysis server 300 analyzes the meaning andcontent of the operation action data through the living actionrecognition unit 321 (S120). The living action recognition unit 321segments the operation action data into data of a preset unit length andattaches living action meta information to each piece of segmented data.The unit length of the operation action data is defined by apredetermined time T (for example, T=1 min). The segmentation order ofthe operation action data is set as i (i=1 to N).

After segmenting the operation action data into the unit time T inchronological order, the living action recognition unit 321 firstdetermines whether an integrated value of the unit length (T) and theparameter i is smaller than the length (time) of the operation actiondata (S121). If it is determined in step S121 that the integrated valueof the unit length (T) and the parameter i is smaller than the length(time) of the operation action data, the living action recognition unit321 attaches living action meta information to the segmented databetween time T*i and time T*(i+1) (step S122). The symbol “*” indicatesintegration processing. In step S122, the meaning and content (livingaction meta information) applicable to the segmented data at the timecan be attached by using, for example, ruled-based branching processing.Alternatively, living action meta information can also be attached thesegmented data using machine learning such as the Hidden Markov Model(HMM) or Neural Network. The number of pieces of living action metainformation attached to the segmented data is not limited to one and aplurality of pieces of living action meta information may be attached.

When living action meta information is attached to the segmented data ofthe operation action data in step S122, the living action recognitionunit 321 adds 1 to the parameter i (S123) to repeat the processing fromstep S121. If it is determined in step S121 that the integrated value ofthe unit length (T) and the parameter i is equal to or greater than thelength (time) of the operation action data, the living actionrecognition unit 321 outputs each piece of segmented data to whichliving action meta information is attached by the processing in stepS122 as living action data (S124). An output result of the living actionrecognition unit 321 may be recorded in a predetermined storage unit(not shown) or may be output directly to the functional unit (in thiscase, the hierarchical structure judgment unit 322) that performs thenext processing.

To return to the description of FIG. 8, when living action data asinformation in which living action meta information is attached tosegmented data constituting operation action data by the living actionrecognition unit 321 is generated, the hierarchical structure judgmentunit 322 attaches hierarchical information representing a hierarchicalrelationship about the meaning and content to the living action data.The processing will be described based on FIG. 10. The hierarchicalstructure judgment unit 322 first determines whether the input livingaction data satisfies at least one of conditions that no hierarchicalinformation is attached and it is possible to further shift to a higherlevel from the attached hierarchical information (S131).

If none of these conditions is satisfied in step S131, that is, thehighest hierarchical information is already attached, the hierarchicalstructure judgment unit 322 terminates the processing shown in FIG. 10.On the other hand, if one of the above conditions is satisfied in stepS131, the hierarchical structure judgment unit 322 combines adjacentsegmented data of the same action of each piece of segmented dataarranged in chronological order (S132). In step S132, processing toregard data discretized in step S120 as one operation action when thesame action continues is performed. A chunk (segment) generated bycombining segmented data may be recorded in a predetermined storage unit(not shown) (S133).

Next, if the chronological arrangement order of segments generated instep S132 is set as j (j=1 to M), the hierarchical structure judgmentunit 322 determines whether j is smaller than the number of segmentsgenerated in step S131 and dictionary information about the action ofthe j-th segment can be acquired (S134). If both of these conditions aresatisfied in step S134, the hierarchical structure judgment unit 322selects and attaches dictionary information optimum to the relevantsegment of the acquired dictionary information (S135). Then, thehierarchical structure judgment unit 322 determines whether still higherhierarchical information can be attached to the segment (j) based on theselected dictionary information and temporal context (S136). If it isdetermined in step S136 that higher hierarchical information can beattached, the hierarchical structure judgment unit 322 attaches higherhierarchical information to the segment (j) and adds 1 to the parameterj (S137). Then, the processing in step S134 and thereafter is repeated.

On the other hand, if it is determined in step S134 that j is equal toor greater than the number of segments generated in step S132 ordictionary information about the action of the j-th segment cannot beacquired, the hierarchical structure judgment unit 322 repeats theprocessing in step S131 and thereafter. Also when it is determined instep S136 that higher hierarchical information cannot be attached to thesegment (j), the hierarchical structure judgment unit 322 repeats theprocessing in step S131 and thereafter.

As shown in FIG. 10, the hierarchical structure judgment unit 322segments segmented data generated in step S120 by piecing togethersegmented data as the same action that is temporally consecutive. Then,related dictionary information is acquired and attached to each segmentand whether still higher hierarchical information can be added isjudged. Thus, by performing the processing in steps S131 to S137, anaction segment (unit segment) in the minimum unit to which hierarchicalinformation is attached is generated.

FIGS. 11 and 12 show a flow chart showing an example of action segmentgeneration processing. As shown in FIG. 11, the hierarchical structurejudgment unit 322 determines whether “another action” sandwiched betweenthe same action of segmented data is present (S200). If “another action”sandwiched between the same action is present, “another action” ismerged into the action before and after “another action” (S202).However, merge processing of “another action” sandwiched betweenoperation actions of “walking” whose precision is sufficiently high andwhich is likely to be a point of change is not performed. After “anotheraction” is merged in step S202 or there is no “another action”sandwiched between the same action in step S200, the hierarchicalstructure judgment unit 322 merges the same actions that are consecutive(S204).

Then, the hierarchical structure judgment unit 322 sets the parameter k(k=1 to K) representing the chronological order of segments generated bymerge processing to the initial value 1 (S206) and determines whetherthe action time of the segment (k) is shorter than a predetermined timeT1 (for example, T1=3 min) (S208). If the action time of the segment (k)is shorter than the predetermined time T1 in step S208, the hierarchicalstructure judgment unit 322 accumulates the segment in a buffer (S210).On the other hand, if the action time of the segment (k) is determinedto be equal to or longer than the predetermined time T1 in step S208,the hierarchical structure judgment unit 322 further determines whetherthe action time of the segment (k) is shorter a predetermined time T2(T2>T1; for example, T2=10 min) (S212).

If the action time of the segment (k) is determined to be shorter thanthe predetermined time T2 in step S212, the hierarchical structurejudgment unit 322 merges the segment (k) into the action immediatelybefore (S214). On the other hand, if the action time of the segment (k)is equal to or longer than the predetermined time T2 in step S212, thehierarchical structure judgment unit 322 decides the operation action ofthe segment as “another action” (S216). Then, the hierarchical structurejudgment unit 322 determines whether the processing of steps S208 toS216 has been performed for all segments (S218) and if there is anyunprocessed segment, the hierarchical structure judgment unit 322 adds 1to k (S220) and then repeats the processing in step S208 and thereafter,

On the other hand, if the processing of steps S208 to S216 has beenperformed for all segments, as shown in FIG. 12, processing to mergeconsecutive segments of the same action is performed (S222). Then, if avehicle action of only the data unit time (for example, only 1 min) ispresent, the action of the segment is set as “another action” (S224).

Next, the hierarchical structure judgment unit 322 determines whetheraction content of the segment is “walking” (S226) and, if the actioncontent is other than “walking”, accumulates the segment in the buffer(S228). On the other hand, if the action content of the segment is“walking”, the hierarchical structure judgment unit 322 determineswhether any vehicle action is accumulated in the buffer (S230). If avehicle action is accumulated in the buffer, the hierarchical structurejudgment unit 322 sets the operation action of the segment as an actionof vehicle with the maximum share from “walking” (S323). On the otherhand, if no vehicle action is accumulated in the buffer, If a vehicleaction is accumulated in the buffer, the hierarchical structure judgmentunit 322 sets the operation action of the segment as “another action”(S324).

Here, hierarchical information added to an action segment will bedescribed based on FIG. 13. As shown in FIG. 13, it is assumed that theoperation content of six unit segments (SG01 to SG06) is “train”,“train”, “walking”, “walking”, “train”, and “train”. The unit segmentSG01 and the unit segment SG02 are merged into an action segment SG07,the unit segment SG03 and the unit segment SG04 are merged into anaction segment SG08, and the unit segment SG05 and the unit segment SG06are merged into an action segment SG09 by the processing shown in FIG.10. The state is set as the grain size 1-0. The action segments SG07 toSG09 in the grain size 1-0 becomes an action segment SG17 of “got on atrain on some line (action B)”, an action segment SG18 of “trainschanged (action C)”, and an action segment SG19 of “got on a train onsome line (action C)” respectively. The grain size at this point is setas 1-1. If the action segments SG17 to SG19 are further merged from thegrain size 1-1, one action segment SG10 of “got on a train (action A)”is obtained. The grain size at this point is set as 1-2.

An action segment holds hierarchical information by being attached tothe action segment combining unit segments or by being attached to theunit segments. When hierarchical information is attached to an actionsegment combining unit segments, it is assumed that, for example, anaction segment SG17 of the action B in FIG. 13 is focused on. The actionsegment SG17 is considered to be an action segment newly generated bycombining the unit segments SG01 and SG02. In this case, the facts thatthe action segment has the grain size 1-1 and the action content is “goton a train on some line”, and the start time and the end time of theaction are attached to the action segment SG17 as hierarchicalinformation. Alternatively, the fact that the action segment SG17includes the unit segments SG01 and SG02 may be attached to the actionsegment SG17 as hierarchical information.

On the other hand, when hierarchical information is attached to unitsegments as action segments of the minimum unit, it is assumed that, forexample, an action segment SG01 in FIG. 13 is focused on. In this case,the facts that the action segment is a unit segment and is contained inthe action segment SG07 in the grain size 1-0, in the action segmentSG17 in the grain size 1-1, and in the action segment SG10 in the grainsize 1-2 is attached to the action segment SG01 as hierarchicalinformation. More specifically, information in which operation contentis associated in the order of hierarchy like, for example, [train,action B, action A] can be represented as hierarchical information.

Hierarchical information may be attached in any form and can also beattached in other forms. The case of attaching hierarchical informationto an action segment combining unit segments is superior in terms of theamount of data and the case of attaching hierarchical information tounit segments is superior in terms of a database search.

Returning to the description of FIG. 8, when action segments to whichhierarchical information is attached from living action data aregenerated in step S130, the living action recognition unit 321 outputseach action segment to the data management unit 330 (S140). The datamanagement unit 330 records the acquired action segment in a storageunit (the unit data storage DB 334 or the hierarchical informationattached data storage DB 335).

[3-4. Processing Content of Action Representation Generation Unit]

The analysis server 300 can accumulate an action log by action segmentsin real time and at the same time, can generate an action representationbased on the meaning and content of an operation action. The analysisserver 300 can also generate an action representation from a past actionhistory. A detailed configuration of the action representationgeneration unit 320 and the data management unit 330 of the analysisserver 300 is shown in FIG. 14. FIG. 14 is functional block diagramshowing the functional configuration of the analysis server 300.

As shown in FIG. 14, the action representation generation unit 320 caninclude, in addition to the living action recognition unit 321 and thehierarchical structure judgment unit 322 described above, a hierarchicalprocessing unit 323, a registration processing unit 324, a commentcreation unit 325, and an acquisition unit 326.

The hierarchical processing unit 323 performs subsequent processing of ajudgment result of the hierarchical structure judgment unit 322. Thehierarchical processing unit 323 functions based on a hierarchicalstructure when only a portion of data to be attached to action segmentsis recorded in the storage unit for slimming down or speedup of data orhierarchical information of the specified action segment is delivered toan application.

As described above, hierarchical information may be attached to anaction segment combining unit segments or to the unit segments. Whenhierarchical information is attached to a combined action segment, thehierarchical processing unit 323 processes the action segment of thehierarchical information selected by the user via the input unit 160. Onthe other hand, when hierarchical information is attached to unitsegments, the hierarchical processing unit 323 generates an actionsegment by combining unit segments based on the hierarchical informationselected by the user via the input unit 160. The hierarchical processingunit 323 a processing result of information to the registrationprocessing unit 324 and the comment creation unit 325.

The registration processing unit 324 records the action segmentgenerated by the hierarchical processing unit 323 in the data managementunit 330. The registration processing unit 324 outputs an action segmentto the data acquisition unit 331 to record the action segment in thehierarchical information attached data storage DB 335.

The comment creation unit 325 creates and attaches a comment such as themeaning and content of an action to a generated action segment. Acomment created by the comment creation unit 325 is output to the dataacquisition unit 331. The data acquisition unit 331 associates thecomment with the corresponding action segment and records the commentin, for example, the hierarchical information attached data storage DB335.

The acquisition unit 326 acquires a predetermined action segment fromthe unit data storage DB 334 or the hierarchical information attacheddata storage DB 335. When, for example, processing that needs to use apast action log is performed by the action representation generationunit 320, the acquisition unit 326 past data recorded in the unit datastorage DB 334 or the hierarchical information attached data storage DB335. Data to be acquired is decided based on instructions from the user.

<4. Action Recording and Display Application>

As described above, by analyzing operation action data acquired by theaction recording device 100 by the analysis server 300, an actionsegment to which the meaning and content of an operation action isattached is generated. An application function to represent an actionlog of the user using action segments will be described below.

[4-1. Representation of Action Log Based on Action Segment]

First, the representation of an action log using action segments will bedescribed. An example of the action log is shown in FIG. 15. The actionlog in FIG. 15 can be displayed, for example, in the display unit 150 ofthe action recording device 100.

An action log includes action segments arranged in chronological order.For each action segment, for example, the start time and the end time ofthe operation action and operation content are displayed. When theoperation content is a movement action like “movement by train”, aposition representation like, for example, from the start location tothe goal (for example, “from Gotanda to Ohsaki”) is added to theoperation content. When the operation content is other than a movementaction like “work” and “meal”, the location (for example, “in Ohsaki”)where the operation is performed is added to operation content.

Further, to notify the user of the operation of such an action segmentin an easy-to-understand manner, an object of operation content may bedisplayed or an object showing the feeling of the user when performingthe operation may also be displayed. Also, the number of steps (stepvalue) for the user to perform the operation or a value (exercise value)indicating energy consumption by the operation may be displayed. Thecontent displayed in each action segment constituting an action log isnot limited to the example of FIG. 15 and information obtained fromsensor information acquired by sensors may further be displayed in eachaction segment.

In the action log shown in FIG. 15, there are locations where times ifadjacent action segments are not continuous. For example, there is a gapbetween the action of “meal at Ohsaki” started at 12:30 and the actionof “was in Shibuya” started at 14:30. In this manner, the action log maybe prevented from displaying all action segments. The non-display of anaction segment may be caused by, for example, editing/deletion of theaction segment by the user or by setting a non-display filter to preventa portion of action segments from being displayed.

As the non-display filter, for example, a filter that prevents thedisplay when the action time is short or a filter that prevents thedisplay of an action segment judged to be unimportant to the use can beconsidered. Also, a filter that prevents the display when the precisionof recognition is low or a filter that allows the display of an actionor location specified by the user may be set.

[4-2. Browsing Action]

When an action log display application is activated in a browsingterminal (for example, the action recording device 100), for example,the user can browse the action log of the user in predetermined units,for example, in units of days. FIG. 16 shows a display example of theaction log when the action log display application is activated. FIG. 16shows a state in which an action log 410 in units of days is displayedin the display unit 150 of the action recording device 100.

The action log 410 includes action segments 412 arranged inchronological order, for example, from the upper end of the screentoward the lower end. In each of the action segments 412, as describedin FIG. 15, the location where an operation is performed, description ofthe location, type of the action, number of steps by the user for theoperation, exercise value and the like are displayed. If the action logfor one day cannot be displayed in the screen of display, the displayunit 150 can be caused to display an action segment that is notdisplayed by performing a screen scroll operation.

By touching a Prev button 422 to display the action log of the previousday of the action log currently displayed or a Next button 424 todisplay the action log of the next day of the action log currentlydisplayed, the display unit 150 can be caused to display an action logof another day. If the action log of the previous day is not presentwhen the Prev button 422 is pressed, the display unit 150 may be causedto display an action log of the day when an action log is acquired nextby further going back to the past. Similarly, if the action log of thenext day is not present when the Next button 424 is pressed, the displayunit 150 may be caused to display an action log of the day when anaction log is acquired next by further moving to the present. Operationbuttons 430 for browsing, editing and other operations of the action log410 are displayed in the display unit 150 and the user can touch thebutton corresponding to desired processing to perform the processing.

If, for example, a calendar button 434 is touched, as shown in FIG. 17,a calendar 440 is displayed in the display unit 150. The calendar 440displayed first in the display unit 150 after the screen transition maybe the current month or the month corresponding to the action log 410displayed before the screen transition. In the calendar 440, a day forwhich the action log 410 can be displayed, that is, a day for which theaction log 410 is accumulated as data can be selected by, for example, atouch operation. When some day is selected from the calendar 440, theaction log 410 of the day is displayed in the display unit 150.

The display of the calendar 440 is changed by a previous (<) button 442or a next (>) button 444. If the previous (<) button 442 is operated,the calendar of the previous month is displayed and if the next (>)button 444 is operated, the calendar of the next month is displayed. Ifno action log of the previous month is present when the previous (<)button 442 is pressed, the calendar 440 of a month when any action logis acquired next may be displayed by further going back to the past.Similarly, if no action log of the next month is present when the next(>) button 444 is pressed, the calendar 440 of a month when any actionlog is acquired next may be displayed by further moving to the present.

FIG. 17 shows an example in which the calendar 440 in units of months isdisplayed, but the present technology is not limited to such an exampleand can display a calendar, for example, in units of weeks, two weeks,or years. In addition, an icon indicating a representative operationaction for the day may be displayed for all days for which an action logcan be displayed in the calendar 440. Accordingly, the user canrecognize the action of the day at a glance. By using such a calendarfunction, the action log 410 desired to browse can easily be searchedfor and the display unit 150 can be caused to display the action log. Tocause a transition from the display screen of the calendar 440 to theprevious screen, a Back button 446 may be operated.

If, for example, a map button 431 of the operation buttons 430 istouched, the action log display application activates a map 450 todisplay position information corresponding to the action log 410 in themap 450. FIG. 18 shows a display example when the map 450 is caused todisplay position information corresponding to the action log 410 byoperating the map button 431.

If, when the action log 410 is displayed, the map button 431 is touchedwhile none of the action segments 412 constituting the action log 410 isselected, for example, a history of all position information of theaction log 410 of the day is displayed on the map 450. If the user is onthe move, a movement locus thereof is displayed on the screen.

On the other hand, if, when the action log 410 is displayed, the mapbutton 431 is touched while one action segment 412 a is selected fromthe action log 410, a history of position information of the actionsegment 412 a is displayed on the map 450. If, for example, as shown onthe left of FIG. 18, the action segment 412 a of “movement fromSaginomiya to Ohsaki by train” is selected and the map button 431 istouched, as shown on the right of FIG. 18, a movement locus by trainfrom the start location (Saginomiya) to the goal (Ohsaki) is displayedon the map 450. Accordingly, on which line the user moved can bepresented in an easy-to-understand manner. While position information isdisplayed in the map 450, time information 452 of the positioninformation may be displayed in the display unit 150.

If the user is not on the move, an icon or the like may be displayed ina location where the operation is performed. In FIG. 18, a case when theone action segment 412 is selected is described, but the presenttechnology is not limited to such an example and a plurality of theaction segments 412 can be selected and position information of all theselected action segments 412 can be displayed on the map 450. In thiscase, the position information on the map 450 may be distinguished bycolor-coding or the like for each action segment and displayed. To causea transition from the display screen of the map 450 to the previousscreen, a Back button 454 may be operated.

[4-3. Correcting Action]

In the foregoing, the method of displaying the action log 410 in thedisplay unit 150 using the action segments 412 analyzed and generated bythe analysis server 300 has been described. However, display content ofthe generated action log 410 may be erroneous. In such a case, the usercan correct content of the action log 410. The correction feedback isreflected in action recognition determination processing. First, themethod of correcting the action log 410 will be described based on FIGS.19 and 20. FIG. 19 is an explanatory view showing a state in which acorrection screen to correct the action segment to be corrected isdisplayed. FIG. 20 is an explanatory view showing an example of thecorrection screen to correct an operation action.

To correct content of the action segment 412, the user selects theaction segment 412 a to be corrected from the action log 410 displayedin the display unit 150 and touches an edit button 435. Then, as shownon the right of FIG. 19, a correction screen 460 to correct the actionsegment 412 a to be corrected is displayed. If, for example, thecorrection of the action segment 412 a of “movement from Saginomiya toOhsaki by train” is selected, a screen transition occurs from thedisplay screen of the action log 410 to the correction screen 460 inwhich the operation content, start location and end location of theoperation, and feeling during operation can be edited.

The operation content can be corrected in an operation contentcorrection area 461 of the correction screen 460. If, for example, theoperation content correction area 461 is selected, as shown in FIG. 20,an operation content candidate list 461 a in which operation contentcandidates for correction are listed is displayed. The user can correctthe operation content by selecting the correct operation content fromthe operation content candidate list 461 a. When “Free input” isselected from the operation content candidate list 461 a, an input fieldinto which the user can freely input operation content is displayed andthe correct operation content can be input.

After selecting operation content from the operation content candidatelist 461 a, the user continues to correct the start location and endlocation of the operation. At this point, if the selected operationcontent is a movement action like, for example, “movement by bus”,corrections of a start location correction area 462, a start locationdescription correction area 463, an end location correction area 464, oran end location description correction area 465 can be made.

A location name list may be displayed for the start location correctionarea 462 and the end location correction area 464 so that the user canselect and input the location name or the user may be enabled todirectly input the location name. In the location name list, forexample, location names to be a landmark such as a building name,station name, or shop name may be displayed. If there is no location tobe a landmark, place names (addresses) may be displayed in the locationname list.

In addition the display unit 150 may be caused to display a startlocation map 462 a and an end location map 464 a that display a map bybeing linked to input content of the start location correction area 462and the end location correction area 464. The start location map 462 aand the end location map 464 a can be caused to display a map of anylocation by a scroll operation on the map. When a touch operation isperformed on a map displayed on the start location map 462 a or the endlocation map 464 a, the location name corresponding to the positionwhere the touch operation is performed may automatically be input intothe start location correction area 462 or the end location correctionarea 464.

The start location description correction area 463 and the end locationdescription correction area 465 are areas where what kind of locationthe location input into the start location correction area 462 and theend location correction area 464 is for the user is input respectively.When the start location description correction area 463 or the endlocation description correction area 465 is touched, for example, asshown in FIG. 20, a description candidate list 463 a or 465 a isdisplayed. The user can input a location description by selecting thecorrect description from the description candidate list 463 a or 465 a.

As the description content of a location, for example, “location to goback to” like the home, “location to work” like a company, and “locationto learn” like a school can be cited. By inputting the description ofsuch a location, what king of meaning the location has for the user canbe grasped and a contribution can be made to improve the precision ofaction recognition for the user. If no correct description is found inthe description candidate list 463 a or 465 a, a description maydirectly be input into the start location description correction area463 or the end location description correction area 465.

When the operation content selected from the operation content candidatelist 461 a in FIG. 20 is an action other than a movement action like,for example, “shopping” or “work”, the end location correction area 464and the end location description correction area 465 can be corrected.The correction method is as described above.

An object indicating the feeling of the user when an operation isperformed can be corrected by, for example, as shown in FIG. 19, afeeling correction area 466. The feeling correction area 466 includes a“no feeling setting” button and a feeling selection button that stepwiseselects good or bad of feelings. The feeling selection button can beconfigured so that, for example, five levels of feeling of “very bad(irritated or depressed)”, “bad (somewhat irritated)”, “not bad (noproblem)”, “good (a bit fine)”, and “very good (refreshing, bracing,full)” are selectable. The user can select the feeling when an operationis performed from the feeling correction area 466.

When all corrections are completed, correction content can be reflectedin the action segment 412 a by pressing a save button 468 a at thebottom or a save button 468 b at the top of the correction screen 460.When the save button 468 a or 468 b is pressed, a transition to thescreen before the transition to the correction screen 460 occurs. When atransition to the screen before the transition to the correction screen460 should occur without reflecting input content in the correctionscreen 460, a cancel button 467 a at the bottom or a cancel button 467 bat the top of the correction screen 460 may be pressed.

[4-4. Combining Actions]

According to the present technology, the display of the action log 410can easily be changed not only by correcting content of each of theaction segments 412, but also by correcting relationships between theaction segments 412. For example, a plurality of the action segments 412may be combined to display the resultant segment as the one actionsegment 412. The combination of the action segments 412 is a function tocombine the plurality of the action segments 412 that are temporallyconsecutive into the one action segment 412. The time range of thecombined action segment 412 extends from the start time of the oldestaction segment 412 to the end time of the newest action segment 412.

FIG. 21 shows an example of the method of combining the action segments412. First, if a merge button 432 is pressed while the action log 410 isdisplayed, a state (action combination mode) in which the actionsegments 412 can be combined is entered. The selected action segments412 can be combined by selecting the action segment 412 to be combinedin action combination mode.

The action of an action segment 412 b selected first among the actionsegments 412 to be combined can be set as operation content after thecombination. In FIG. 21, the operation content of “work in Ohsaki”becomes the operation content after the combination. Then, the otheraction segments 412 to be combined are selected. The selection of theaction segment 412 can be made by a touch operation or a drag operation.In FIG. 21, action segments surrounded by a reference sign 412 c areselected for combination. Then, if the save button 468 a is pressedafter the action segments 412 to be combined are selected, the actionsegments 412 to be combined are displayed by being combined. Thecombined action segment 412 represents the action of “work in Ohsaki” isdone between 9:55 and 22:42. To cancel the combination mode, the cancelbutton 467 a may be pressed.

In the example of FIG. 21, the operation content of the action segment412 after the combination is decided in favor of the action segment 412selected first, but the present technology is not limited to such anexample. An example of another method of deciding operation content ofthe action segment 412 after combination is shown in FIG. 22. In FIG.22, an action segment to be combined is selected by a drag operation.That is, the first action segment (start segment) to be combined isfirst selected by contact with a finger and the finger is moved while incontact until the last action segment (end segment) to be combined isselected. In this manner, the action segments (action segmentssurrounded by the reference sign 412 c) to be combined are decided.Then, when the save button 468 a (see FIG. 21) is pressed, the actionsegment after the combination is displayed.

For example, the operation content of any action segment to be combinedmay be set as the operation content of the action segment after thecombination. If, for example, the operation content of “walked inOhsaki” is selected, the operation content of an action segment 412 d 1after the combination becomes “walked in Ohsaki”.

The operation content of the action segment after the combination may bedecided, for example, by majority of action segments to be combined. Inthe example of FIG. 22, for example, among the four action segments 412to be combined, the operation content of the two action segments 412 is“work in Ohsaki”, the operation content of the one action segment is“meal in Ohsaki”, and the operation content of the other one actionsegment is “walked in Ohsaki”. Therefore, the operation content of “workin Ohsaki” of the most action segments is decided as the operationcontent of the action segment 412 d 2 after the combination.

Alternatively, the operation content of the action segment after thecombination may be decided, for example, by reanalyzing action segmentsto be combined. In the example of FIG. 22, for example, the operationcontent of the four action segments 412 to be combined includes “work inOhsaki”, “meal in Ohsaki”, and “walked in Ohsaki”. Accordingly, themeaning and content of user's actions are reanalyzed and, for example,the operation content of “was in Ohsaki” can be decided as the operationcontent of an action segment 412 d 3 after the combination.

Therefore, the action segments 412 can easily be combined by selectingthe action segments to be combined.

[4-5. Dividing Action]

Also according to the present technology, for example, a plurality ofthe action segments 412 can be divided and displayed as a plurality ofthe action segments 412 as a correction of the relationship between theaction segments 412. The division of the action segment 412 is afunction to segment the one action segment 412 into a plurality of theaction segments 412. As the division method of the action segment 412,for example, a method of setting the time to divide the action segment412 and a division method using hierarchical information are known.

For example, FIG. 23 shows an example of a division method by timesettings of the action segment 412. First, if a division 433 is pressedwhile the action log 410 is displayed, a state (action division mode) inwhich the action segment 412 can be divided is entered. The selectedaction segment 412 can be divided at a specified time by selecting theaction segment 412 to be divided in action division mode.

For example, it is assumed that an action segment 412 e is selected fordivision in FIG. 23. Then, a division time setting screen to input thedivision time of the action segment 412 e to be divided is displayed. Inthe division time setting screen, ant time between the start time andthe end time of the action segment to be divided can be set. Then,pressing the save button 468 a divides action segment 412 to be dividedand displays the divided action segments 412. If, for example, thedivision time is set to 11:50, the action segment 412 is divided into afirst action segment of “work in Ohsaki” between 9:58 and 11:50 and asecond action segment of “work in Ohsaki” between 11:50 and 12:29. Tocancel the division mode, the cancel button 467 a may be pressed.

Also, for example, FIG. 24 shows an example of a division method basedon hierarchical information of the action segment 412. It is assumedthat the action division mode is already set in FIG. 24. If the userselects the action segment 412 e to be divided while the action log 410is displayed, as shown in FIG. 24, a hierarchical list 414 representingthe action segment 412 e to be divided by more detailed action segmentsis displayed. When the hierarchical list 414 is displayed, the userselects the action segment to be the division point.

If, for example, an action segment 414 a is selected in FIG. 24, forexample, the end time of the action segment 414 a is decided as thedivision time and, as shown on the right of FIG. 24, the action segmentis divided into a first action segment 412 e 1 of “work in Ohsaki”between 9:30 and 11:59 and a second action segment 412 e 2 of “work inOhsaki” between 11:59 and 12:30.

[4-6. Representation of Action Segment Based on Segmentation Grain Size]

In the present technology, the action segments 412 constituting theaction log 410 hold a hierarchical relationship based on the meaning andcontent thereof as hierarchical information. The display roughness ofthe displayed action log 410 can be changed by changing the segmentationgrain size using the hierarchical information. The display roughness canbe changed by using, for example, a slider or a zoom button.

FIG. 25 shows a case when display roughness is changed by using a slider471. A knob 472 to set the display roughness (grain size) is displayedin the slider 471 and the display roughness can be changed by changingthe position of the knob 472. When, for example, as shown on the left ofFIG. 25, the knob 472 is positioned on the side of small grain size ofthe slider 471, the action log 410 is displayed by the detailed actionsegments 412. As the knob 472 is moved toward the side of large grainsize of the slider 471 from the above state, a plurality of the actionsegments 412 are displayed by being combined based on the segmentationgrain size attached to the action segments 412 in advance.

FIG. 26 shows a case when the display roughness is changed by using azoom button 473. A plurality of buttons 474 to set the display roughness(grain size) is displayed in a row in the zoom button 473. By checkingone of the plurality of buttons 474, the action log 410 can be made tobe displayed in the display roughness corresponding to the button 474.When, for example, as shown on the left of FIG. 26, the button 474 onthe side of small grain size in the zoom button 473 is checked, theaction log 410 is displayed by the detailed action segments 412. As thebutton 474 on the side of large grain size is checked from the abovestate, a plurality of the action segments 412 are displayed by beingcombined based on the segmentation grain size attached to the actionsegments 412 in advance.

Thus, the display roughness of the action log 410 can easily be changedbased on the segmentation grain size attached to the action segment 412so that the user can view the action log 410 in the desired displayroughness.

The display roughness of the action log 410 is changed in FIGS. 25 and26, but in the present technology, the display roughness of the actionsegment 412 can also be changed depending on the purpose. That is, thedisplay roughness of the action segment 412 is changed independently ofthe type of action. Work, shopping, movement and the like can beconsidered as the types of action and, for example, by checking a button476 of the corresponding action from an action type selection list 475as shown in FIG. 27, the display roughness of only the action can bechanged.

When, for example, as shown in FIG. 27, a button 476 b of “work detail”is checked in the action type selection list 475, action segments 412 f1, 412 g 1 related to work are displayed in detail. For example, theoperation content of “work in Ohsaki” for the action segment 412 f 1 isdisplayed by five action segments 412 f 2 of “desk work in Ohsaki”,“movement on foot”, “meeting in Ohsaki”, “movement on foot”, and “deskwork in Ohsaki”. Incidentally, the operation content of the actionsegment 412 g 1 is the most detailed and thus, the same content isdisplayed after the display roughness is changed (action segment 412 g2).

When, for example, as shown in FIG. 28, a button 476 c of “shoppingdetail” is checked in the action type selection list 475, an actionsegment 412 h 1 related to shopping is displayed in detail. For example,the operation content of “shopping in Shibuya” for the action segment412 h 1 is displayed by seven action segments 412 h 2 of “shopping inShibuya”, “movement on floor”, “shopping in Shibuya”, “movement betweenshops in Shibuya”, “shopping in Shibuya”, “moving between shops inShibuya”, and “shopping in Shibuya”.

Further, when, for example, as shown in FIG. 29, a button 476 d of“movement detail” is checked in the action type selection list 475,action segments 412 i 1, 412 j 1, 412 k 1 related to movement aredisplayed in detail. For example, the operation content of “movementfrom Saginomiya to Ohsaki by train” for the action segment 412 i 1 isdisplayed by five action segments 412 i 2 of “waiting for train inSaginomiya”, “movement from Saginomiya to Takadanobaba by train”,“trains changed in Takadanobaba”, “movement from Takadanobaba to Ohsakiby train”, and “movement in Ohsaki on foot”. Similarly, the operationcontent of “moving from Ohsaki to Shibuya by train” for the actionsegment 412 j 1 is displayed by four action segments 412 j 2 of“movement to Ohsaki station”, “waiting for train in Ohsaki station”,“movement from Ohsaki to Shibuya by train”, and “movement to Shibuya”.For the action segment 412 k 1, similarly detailed content is displayed.

When the action segments 412 should be displayed in the same displaygrain size regardless of the action, for example, as shown in FIG. 30, abutton 476 a of “uniform detail” may be checked in the action typeselection list 475. Accordingly, all the action segments 412 of theaction log 410 are displayed in detail in the same grain size.

Thus, because the display roughness of the action segments 412 can bechanged independently in accordance with the type of action, only theaction the user wants to check in detail can be displayed in detail.

Incidentally, the method of changing the display roughness shown inFIGS. 25 and 26 and the method of changing the display roughness inaccordance with the type of action shown in FIGS. 27 to 30 may becombined. For example, as shown in FIG. 31, a slider may be provided foreach type of action so that the display roughness of each type of actioncan be adjusted. FIG. 31 is shows a display grain size setting unit 477provided with a slider 478 a to set the display roughness of an actionsegment related to “work”, a slider 478 b to set the display roughnessof an action segment related to “shopping”, and a slider 478 c to setthe display roughness of an action segment related to “movement”. Bymoving respective knobs 479 a, 479 b, 479 c of the sliders 478 a, 478 b,478 c of the display grain size setting unit 477, the display roughnesscan be adjusted for each type of action.

[4-7. Deleting Action]

According to the present technology, the action segment 412 can bedeleted from the action log 410. If, for example, as shown in FIG. 32,an action segment 4121 to be deleted is selected and a deletion button436 is pressed while the action log 410 is displayed, as shown on theright of FIG. 32, a deletion confirmation screen 480 is displayed. Inthe deletion confirmation screen 480, the user can be caused to enterthe reason for deleting the action segment 4121 to be deleted. When theuser presses a button on which the reason for deletion is written, theaction segment 4121 to be deleted is deleted from the action log 410.Depending on the reason selected by the user, the deletion of the actionsegment may be fed back as an action correction.

[4-8. Posting Action]

According to the present technology, content of the action segment 412of the action log 410 can be posted. If, for example, as shown in FIG.33, an action segment 412 m to be posted is selected and a post button437 is pressed while the action log 410 is displayed, as shown on theright of FIG. 33, a posting screen 482 is displayed. In the postingscreen 482, the operation content of the action segment 412 m to beposted is automatically pasted to a posting content input area 482 a.When the user presses a posting button 482 b, the description content inthe posting content input area 482 a is posted to a posting site.

[4-9. Action Log Acquisition Stop Processing]

According to the present technology, when the acquisition of an actionlog should be stopped for some reason, for example, as shown in FIG. 34,a settings screen 490 is made to display by pressing a settings button438. In the settings screen 490, various settings about the action logdisplay application can be made. When, for example, the acquisition ofthe action log 410 should be stopped, “stop” of an acquisition functionsetting unit 491 that sets the operation of the action log acquisitionfunction is selected. Accordingly, the action log display applicationstops the action log acquisition function. To restart the stopped actionlog acquisition function, “restart” of the acquisition function settingunit 491 may be selected.

[4-10. Updating Display Content]

The action log display application in the present technologyautomatically uploads operation action data acquired by the actionrecording device 100 to the action log server 200 in predeterminedtiming (for example, twice per day). Also, the analysis server 300automatically generates an action segment in predetermined timing (forexample, twice per day). While an action log is displayed based ongenerated action segments, an action log displayed in accordance withthe system function or circumstances may not correspond to the latestresults. Thus, by pressing an update button 493 that updates the actionlog displayed in the settings screen 490 of FIG. 34 to the latestresults, the action log can be updated to the latest results. When atransition to the action log display screen occurs after the updatebutton 493 being pressed, the display unit 150 can be caused to displaythe latest results.

<5. Reflection Processing of Correction Feedback>

In an action log display system in the present technology, the meaningand content of an action is analyzed by the analysis server 300 and anaction log is displayed by action segments. However, as described above,content of the displayed action log may not all correct. Thus, accordingto the present technology, the user can make corrections to correctcontent by using the action log display application. In the presenttechnology, correction feedback of the user is reflected in the nextanalysis processing by the analysis server 300 and used to improve theprecision of the next and subsequent analysis results. The reflectionprocessing of correction feedback will be described below based on FIGS.35 to 42.

[5-1. Properties of Correction Feedback]

In the present technology, the precision of analysis results is improvedby reflecting correction feedback of the user in analysis processing,but the user may not correct all errors of analysis results by theanalysis server 300. That is, content of an action log that is notcorrected may not necessarily be correct. Thus, in the presenttechnology, it is necessary to assume a system capable of collectingsubstantially biased information only. In addition, analysis resultsbefore corrections by the user do not necessarily match the latestanalysis results. Thus, by reflecting information showing which actionsegment is corrected in what way in analysis processing for each user,the action specific to each user can be learned, which is considered toeffectively work to improve the precision of analysis results.

[5-2. Action Recognition Processing]

In consideration of the above points, according to the presentembodiment, an action pattern is decided based on characteristic amountanalysis results in recognition processing of an operation action andacquires a plurality of probability distributions corresponding to theaction pattern, time, and position information (location). In this case,a weight of a histogram is assigned and an operation action isrecognized based on results of assigning weights depending on thelocation. If position information cannot be acquired or there is no needto acquire position information, uniform weights may be assigned orspecific weights like “no location can be acquired” or “there is no needfor location” may be assigned.

FIG. 35 shows an example of action recognition processing by the livingaction recognition unit 321. FIG. 36 shows operation action estimationinformation that decides an operation action. The operation actionestimation information is, for example, information showing the relationbetween a weighting factor depending on the location and the probabilitydistribution of each action and, as shown in FIG. 36, a plurality (fourin FIG. 36) of probability distributions of actions of, for example,“shopping”, “work”, “meal”, and “others” is acquired. Then, theweighting factor depending on the location is set to each probabilitydistribution. The operation action estimation information is preset andrecorded in, for example, the analysis parameter DB 333.

After an action log in the unit time is acquired, the living actionrecognition unit 321 starts processing to recognize the action of theaction log. First, as shown in FIG. 35, at least one pair of theprobability distribution and the weighting factor depending on thelocation is acquired based on the action pattern, time information, andposition information (S300).

Next, in steps S302 to S306, the living action recognition unit 321performs processing to decide operation content of the action log in theunit time. First, it is assumed that the number of pairs of theprobability distribution and the weighting factor acquired in step S300is n and the parameter representing the processing number is i (i=0 ton) (S302). Then, the living action recognition unit 321 multiplies theprobability distribution by the weighting factor of each action for thefirst (i=0) pair of the probability distribution and the weightingfactor (S304). If, for example, in FIG. 36, the first pair is the pairof the probability distribution and the weighting factor in the firstrow, the probability of 50 is multiplied by the weighting factor of 1for “shopping” and the probability of 10 is multiplied by the weightingfactor of 1 for “work”. Then, the probability of 10 is multiplied by theweighting factor of 1 for “meal” and the probability of 30 is multipliedby the weighting factor of 1 for “others”. Accordingly, the integratedvalues (“shopping”: 50, “work”: 10, “meal”: 10, “others”: 30) of actionsare acquired.

When the processing in step S304 is completed, the living actionrecognition unit 321 adds 1 to the parameter i (S306) and repeats theprocessing in step S302 and thereafter. In the example of FIG. 36, theliving action recognition unit 321 multiplies the probabilitydistribution by the weighting factor of each action for the next (i=1)pair of the probability distribution and the weighting factor, that isthe pair of the probability distribution and the weighting factor in thesecond row (S304). For the pair in the second row, first the probabilityof 10 is multiplied by the weighting factor of 6 for “shopping” and theprobability of 50 is multiplied by the weighting factor of 6 for “work”.Then, the probability of 10 is multiplied by the weighting factor of 6for “meal” and the probability of 30 is multiplied by the weightingfactor of 6 for “others”. Accordingly, the integrated values(“shopping”: 60, “work”: 300, “meal”: 60, “others”: 180) of actions areacquired.

Then, the living action recognition unit 321 adds the integrated valuein the second row to the integrated value in the first row for eachaction. This results in integrated values of “shopping”: 110, “work”:310, “meal”: 70, “others”: 210. Similarly, integrated values arecalculated for the pairs of the probability distribution and theweighting factor in the third and fourth rows and these integratedvalues are added to the above integrated values of each action tofinally obtain added values of “shopping”: 260, “work”: 460, “meal”:420, “others”: 460.

The living action recognition unit 321 decides the action of the maximumfinal added value as the operation content of the action log. In theexample of FIG. 36, both “work” and “others” have the maximum addedvalue of 460, but in this case, the operation content is decided bygiving priority to actions other than “others”.

Therefore, in the example of FIG. 36, “work” is recognized as theoperation content.

[5-3. Reflection Processing of Correction Feedback]

(5-3-1. Overview of Reflection Processing of Correction Feedback)

As described based on FIGS. 35 and 36, recognition processing results ofoperation content changes considerably depending on operation actionestimation information showing the relation between the weighting factordepending on the location and the probability distribution. Thus, theaction log is corrected by the user, correction content is reflected inoperation action estimation information recorded in the analysisparameter DB 333 by the feedback adjustment unit 332 of the analysisserver 300. Accordingly, the precision of recognition processing ofoperation content can be enhanced.

An overview of reflection processing of correction feedback will beprovided based on FIG. 37. It is assumed that the relation between theweighting factor depending on the location and the probabilitydistribution of each action shown in FIG. 36 is held as operation actionestimation information before correction. As a result of generating anaction segment from operation action data based on such operation actionestimation information, the user is assumed to input correctioninformation (correction feedback) of the action segment from the inputunit 160. The correction feedback is converted into data in a formatthat can be transmitted to the analysis server 300 by the inputinformation processing unit 144 before being transmitted to the analysisserver 300 via the client interface unit 130.

The analysis server 300 having received the correction feedback from theaction recording device 100 through the analysis server interface unit310 reflects content of the correction feedback in the operation actionestimation information through the feedback adjustment unit 332. At thispoint, the feedback adjustment unit 332 corrects the probabilitydistribution of the operation action estimation information if thecontent of the correction feedback concerns an action and corrects theweighting factor depending on the location if the content of thecorrection feedback concerns position information (location).

It is assumed that, for example, as shown in FIG. 37, the operationcontent of “work” is acquired as an analysis result, but correctionfeedback to change the operation content to “shopping” by the user isreceived. In this case, the feedback adjustment unit 332 corrects, amonga plurality of probability distributions, the probability distributionhaving the maximum probability of “work”. For example, the feedbackadjustment unit 332 makes a correction to set the probability of “work”as an analysis result and the probability of “shopping” as a correctionresult to the average value of these two probabilities for theprobability distribution in the second row with the maximum probabilityof “work”.

It is assumed, on the other hand, that an analysis result of “locationto work” is acquired, but correction feedback to change the locationdescription to “location to do shopping frequently” by the user isreceived. In this case, the feedback adjustment unit 332 corrects, amonga plurality of probability distributions, the weighting factor of theprobability distribution having the maximum probability of “shopping”.For example, the feedback adjustment unit 332 makes a correction ofincreasing the weighting factor in the first row with the maximumprobability of “shopping” by a factor of a predetermined number (forexample, 10).

By correcting the operation action estimation information in thismanner, correction feedback is reflected in analysis results of actionsegments so that the precision of analysis results of the operationcontent can be expected. The reflection processing of correctionfeedback will be described in more detail below based on FIGS. 38 to 42.

(5-3-2. Reflection Processing of Correction Feedback of an Action)

First, the reflection processing of correction feedback of an actionwill be described based on FIGS. 38 to 40. FIG. 38 is a flow chartshowing the reflection processing of correction feedback of an action.FIG. 39 is an explanatory view illustrating corrections of the operationaction estimation information based on the processing in FIG. 38. FIG.40 is a flow chart showing other reflection processing of correctionfeedback of the action. “001”, “002”, and “003” shown in the top row ofeach column in FIG. 39 are IDs representing respective actions. It isassumed in the description below that the analysis result by the livingaction recognition unit 321 is “action 002” and the correct action byuser's correction feedback is “action 003”.

When correction feedback is received from the action recording device100, the feedback adjustment unit 332 first recognizes correctioncontent. It is assumed here that operation content of an action segmentis corrected. The feedback adjustment unit 332 acquires the actionsegment to be corrected from the unit data storage DB 334 or thehierarchical information attached data storage DB 335 and startsprocessing shown in FIG. 38.

The feedback adjustment unit 332 first acquires the probabilitydistribution (partial probability distribution) used to recognize theoperation content of the action segment to be corrected from operationaction estimation information stored in the analysis parameter DB 333(S310). Next, the feedback adjustment unit 332 calculates a value M(i)obtained by multiplying the maximum probability of each probabilitydistribution by the weighting factor of the row for the partialprobability distribution and sorts these probability distributions(S311).

The parameter indicating the order of sorted probability distributionsis set as i (i=0 to n) and the number of probability distributionsconstituting the partial probability distribution is set as n. Then, thefeedback adjustment unit 332 determines whether the parameter i issmaller than n and the multiplied value M(i) is larger than apredetermined threshold th (S312). If the conditions in step S312 arenot satisfied, the processing shown in FIG. 38 is terminated. If, forexample, the operation action estimation information on the left of FIG.39 is provided, only probability distributions of the fourth to sixthrows are corrected. On the other hand, if the conditions in step S312are satisfied, the feedback adjustment unit 332 acquires a correctionratio C(i) from a loss ratio calculation function using an action havingthe maximum value of probability distribution of each action of targetrows from operation action estimation information and the correct actionobtained from correction feedback (S313).

The loss ratio calculation function is assumed to be a singlecomprehensive measure representing losses caused when some availabledecision is made. In the present embodiment, the loss ratio calculationfunction is used to set, for example, a correction ratio tablerepresenting a correction ratio C between the action of analysis resultsand the correct action as shown in the lower portion of FIG. 39. Thecorrection ratio table can be preset and can be stored in the analysisparameter DB 333. The feedback adjustment unit 332 acquires thecorrection ratio C(i) between the action having the maximum value ofprobability distribution and the correct action “action 003” from thecorrection ratio table. If, for example, the processing in step S313 isperformed for the probability distribution in the fourth row of theoperation action estimation information on the left of FIG. 39 isperformed, the correction ratio C(0)=0 between “action 003” having themaximum value of probability distribution and the correct action “action003” is obtained.

Then, the feedback adjustment unit 332 subtracts the correction ratioC(i) acquired in step S313 from the value of the probabilitydistribution of the action of the maximum value of probabilitydistribution, adds the correction ratio C(i) to the value of theprobability distribution of the correct action, and reflects thesecorrections in the operation action estimation information (S314). If,for example, the processing in step S314 is performed for theprobability distribution in the fourth row of the operation actionestimation information on the left of FIG. 39 is performed, theprobability distribution of the row is not changed because thecorrection ratio C(0)=0. Then, the feedback adjustment unit 332 adds 1to the parameter i (S315) and repeats the processing in step S312 andthereafter.

If, for example, the processing in step S313 is performed for theprobability distribution in the fifth row of the operation actionestimation information on the left of FIG. 39 is performed, thecorrection ratio C(1)=5 between “action 002” having the maximum value ofprobability distribution and the correct action “action 003” isobtained. Then, if the processing in step S314 is performed, the valueof the probability distribution of “action 002” is corrected from 50 to45 and the value of the probability distribution of “action 003” iscorrected from 10 to 15 based on the correction ratio C(1)=5.

Similarly, if the processing in step S313 is performed for theprobability distribution in the sixth row of the operation actionestimation information on the left of FIG. 39 is performed, thecorrection ratio C(1)=10 between “action 001” having the maximum valueof probability distribution and the correct action “action 003” isobtained. Then, if the processing in step S314 is performed, the valueof the probability distribution of “action 001” is corrected from 40 to30 and the value of the probability distribution of “action 003” iscorrected from 10 to 20 based on the correction ratio C(1)=10. Byperforming the above processing, the operation action estimationinformation after the correction feedback being reflected as shown onthe right of FIG. 39 can be obtained.

In this manner, the operation content of correction feedback isreflected in the operation action estimation information. The reflectionprocessing shown in FIG. 38 is effective in being able to control thereflection speed or divergence. That is, a more impermissible error canbe reflected earlier and a value can be made to converge if the value ishandled as a ratio to the maximum value. In the reflection processingshown in FIG. 38, the correction ratio C(i) is added to or subtractedfrom the value of the probability distribution, but the presenttechnology is not limited to such an example and, for example,correction feedback may be reflected in operation action estimationinformation by multiplying the value of the probability distribution bythe correction ratio.

In the reflection processing of correction feedback shown in FIG. 38,content of the correction feedback is reflected in operation actionestimation information by using a correction ratio table, but thepresent technology is not limited to such an example. For example, asshown in FIG. 40, a feedback system using a neural network technique maybe configured. It is assumed that operation content of an action segmentis corrected also in FIG. 40.

The feedback adjustment unit 332 first acquires the probabilitydistribution (partial probability distribution) used to recognize theoperation content of the action segment to be corrected from operationaction estimation information stored in the analysis parameter DB 333(S320). Next, the feedback adjustment unit 332 calculates a value M(i)obtained by multiplying the maximum probability of each probabilitydistribution by the weighting factor of the row for the partialprobability distribution and sorts these probability distributions(S321). The processing in steps S320, S321 can be made the same as theprocessing in steps S310, S311 in FIG. 38.

If the parameter showing the order of sorted probability distributionsis set as i(i=0 to n), the feedback adjustment unit 332 determineswhether the parameter i is smaller than n (S322). If the condition instep S322 is not satisfied, the processing shown in FIG. 40 isterminated. On the other hand, if the condition in step S322 issatisfied, the feedback adjustment unit 332 uses the neural networktechnique to reflect correction content of correction feedback in eachprobability distribution of operation action estimation informationbased on the weighting factor (S323). Then, the feedback adjustment unit332 adds 1 to the parameter i (S324) and repeats the processing in stepS322 and thereafter.

Thus, by using, instead of the correction ratio table, learningprocessing such as the neural network technique, content of correctionfeedback can be reflected in each value of operation action estimationinformation without the need to set the correction ratio table inadvance.

(5-3-3. Reflection Processing of Correction Feedback of an Action andPosition Information)

Next, the reflection processing of correction feedback of an action andposition information will be described based on FIG. 41. FIG. 41 is aflow chart showing the reflection processing of correction feedback ofthe action and position information.

The feedback adjustment unit 332 acquires the action segment to becorrected from the unit data storage DB 334 or the hierarchicalinformation attached data storage DB 335 and starts processing shown inFIG. 41. The feedback adjustment unit 332 first determines whethercorrection feedback contains position information (S330). If thecorrection feedback contains position information in step S330,processing in step S331 and thereafter is performed to reflect correctedposition information in operation action estimation information.

In step S331, whether any correction related to position information ismade on action segments accompanied by movement is determined. If acorrection related to position information is made on action segmentsaccompanied by movement, representative coordinates of end points (tworepresentative coordinates like position X to position Y) are calculated(S332). On the other hand, if no correction related to positioninformation is made on action segments accompanied by movement,representative coordinates of the movement are calculated (S333).Incidentally, representative coordinates can be calculated by using thecenter, center of gravity, most frequent point and the like.

Next, the feedback adjustment unit 332 records representativecoordinates calculated in step S332 or S333, the precision, and attachedattributes in a feedback DB (not shown) (S334). The feedback DB is astorage unit provided in the analysis server 300. Then, the feedbackadjustment unit 332 analyses operation content using new positioninformation recorded in the feedback DV in step S334 and determineswhether the analysis result matches the correct action input by thecorrection feedback (S335). If it is determined in step S335 that theoperation content analyzed by using new position information matches thecorrect action, a judgment can be made that correction feedback aboutposition information is correctly reflected and also there is no errorin the action content. Therefore, the feedback adjustment unit 332judges that the reflection processing of correction feedback iscompleted and terminates the processing in FIG. 41.

On the other hand, if it is determined in step S335 that the operationcontent analyzed by using new position information does not match thecorrect action, a judgment can be made that with corrections of positioninformation alone, correction feedback is not correctly determined. Inthis case, processing in steps S336 to S341 is performed to reflectoperation content of the correction feedback in operation actionestimation information. The processing in steps S336 to S341 can be madethe same as the processing in FIG. 38.

That is, the feedback adjustment unit 332 first acquires the probabilitydistribution (partial probability distribution) used to recognize theoperation content of the action segment to be corrected from operationaction estimation information stored in the analysis parameter DB 333(S336). Next, the feedback adjustment unit 332 calculates a value M(i)obtained by multiplying the maximum probability of each probabilitydistribution by the weighting factor of the row for the partialprobability distribution and sorts these probability distributions(S337).

The parameter indicating the order of sorted probability distributionsis set as i (i=0 to n) and the number of probability distributionsconstituting the partial probability distribution is set as n. Then, thefeedback adjustment unit 332 determines whether the parameter i issmaller than n and the multiplied value M(i) is larger than apredetermined threshold th (S338). If the conditions in step S338 arenot satisfied, the processing shown in FIG. 41 is terminated. On theother hand, if the conditions in step S338 are satisfied, the feedbackadjustment unit 332 acquires a correction ratio C(i) from a loss ratiocalculation function using an action having the maximum value ofprobability distribution of each action of target rows from operationaction estimation information and the correct action obtained fromcorrection feedback (S339).

Then, the feedback adjustment unit 332 subtracts the correction ratioC(i) acquired in step S339 from the value of the probabilitydistribution of the action of the maximum value of probabilitydistribution, adds the correction ratio C(i) to the value of theprobability distribution of the correct action, and reflects thesecorrections in the operation action estimation information (S340). Then,the feedback adjustment unit 332 adds 1 to the parameter i (S341) andrepeats the processing in step S338 and thereafter. By performing theabove processing, the operation action estimation information after thecorrection feedback being reflected can be obtained.

Incidentally, instead of the processing in steps S336 to S341, theprocessing shown in FIG. 40 may be performed. Also in this case, theoperation action estimation information after the correction feedbackbeing reflected can similarly be obtained.

The correction feedback of position information may be reflected by, asshown on the lower left of FIG. 37, changing the weighting factor of theprobability distribution of operation action estimation information.Alternatively, a corrected attribute dependence section, a specifiedattribute dependence section, and an ID dependence section may be set toreflect the correction feedback of position information in eachweighting factor. For example, the specified attribute dependencesection is strengthened for the position information of “house” and aperipheral attribute dependence section is strengthened for the positioninformation of “location to do shopping”. For the position informationof, for example, “company”, a plurality of pieces of positioninformation like different offices may be present. In this case, theposition information of the same meaning can correctly be selected bygiving differences like business content and scale to each piece ofposition information as respective features.

The added amount of weighting factor for position information may bedecided based on, for example, original position information or changedfor each attribute type of position information. Further, a probabilitydistribution specific to position information may randomly be generatedand added to operation action estimation information. Accordingly,over-learning can be prevented.

<6. Others>

[6-1. Personal Modeling of Action Pattern]

In an action log display system according to the present embodiment, anaction log is displayed by using action segments to which the meaningand content is attached. By performing, for example, autocorrelationprocessing or filter processing using these action segments, temporal oraction errors can be absorbed. Then, a user's typical action pattern canbe extracted from a small amount of data.

As a functional unit to extract a user's typical action pattern, asshown in FIG. 42, a typical action pattern generation unit 336 isprovided in the analysis server 300. To extract a typical actionpattern, an action log for a predetermined period (for example, for oneday) acquired by the living action recognition unit 321 from the actionlog server 200 is first smoothed and then output to the typical actionpattern generation unit 336. The typical action pattern generation unit336 generates a typical action pattern using a statistical technique,for example, cross correlation processing on action segments of thesmoothed action log.

In the example of FIG. 42, the typical action pattern generation unit336 acquires a typical action pattern of the user as a result ofanalyzing, based on action logs of seven days, correlations of actionsegments of these action logs. Thus, by analyzing action segments towhich hierarchical information represented by the meaning and content ofactions and showing relations between action segments is attached, auser's probable typical action pattern can be generated even from asmall amount of data.

[6-2. Position Display Technique by Moving Medium/Means Determination]

If an action is recognized as a movement action when an action segmentis generated, the living action recognition unit 321 identifies positioninformation of the user based on which medium of transport the user usesto move or which means of transport the user uses to move (FIG. 43).More specifically, when a movement action is recognized from operationaction data, the living action recognition unit 321 analyzes how todisplay position information thereof. In this case, the living actionrecognition unit 321 first acquires nearest station candidates as aneasy-to-use landmark, the last position information and informationrelated thereto. Nearest station candidates can be identified by usingthe line name, station name, distance to the station and the like. Asthe last position information and information related thereto, themedium of transport or means of transport, time difference, distancefrom the last latitude/longitude and the like are acquired.

The living action recognition unit 321 assigns weights to the lines andstations using the above information to identify the nearest station.Weights may be assigned to lines and stations by, for example,increasing weights of nearest station candidates with a decreasingdistance or assigning weights preferentially to lines and stations thatare continuously acquired in action logs. Alternatively, weights may beassigned in consideration of distance differences or time differencesthat can be acquired from information up to the last time. Accordingly,if the fact of being a predetermined distance apart or that apredetermined time has passed is recognized from the information up tothe last time and information this time, the possibility of havingchanged trains to another line can be considered.

(6-2-1. Line Estimation Processing)

The estimation of line can be determined from, for example, the numberof passed stations recognized from an action log. In addition, themovement locus of the user can be estimated by considering thepossibility of changing trains at a station identified from positioninformation or whether a direct service between a plurality of lines isavailable. If a plurality of lines runs between the same stations, whichline is used can be identified by estimating a more likely line from theuser's past movement locus or acquiring more detailed positioninformation from a position information acquisition sensor.

As a result of performing the above line estimation processing, forexample, as shown in FIG. 44, movement of the user can be displayed, forexample, on a map. Incidentally, the line may be displayed on a map onlywhen the precision of line estimation processing of a predeterminedvalue or more is secured.

(6-2-2. Station Name Selection Processing)

The station name is selected by, as described above, identifying thenearest station. In this case, even if the user does not actually move,changes in latitude/longitude may erroneously be recognized due to anerror of radio field intensity of a sensor. Thus, for example, as shownin FIG. 45, the expression may be changed in accordance with theposition precision of the station name of the nearest station. If, forexample, the Ohsaki station is identified as the nearest station, theexpression is changed like “work in the Ohsaki station”, “work near theOhsaki station”, or “work in Ohsaki” based on the distance differencebetween the position of the nearest station and the positioninformation. Accordingly, the location where work is done can beexpressed more appropriately.

If the medium/means of transport is not movement by train, for example,priority may be given to the identified nearest station as a landmark torepresent the location of operation by the station name (excluding“station”). For example, it is assumed that, as a result of analyzing anaction log, movement by car is recognized and the “Higash-Koganeistation” and the “Shin-Koganei station” are identified as landmarks. Inthis case, it is not natural to move between stations by car and thus,action content can naturally be expressed by representing the startlocation and the goal as the “Higash-Koganei station” and the“Shin-Koganei station”.

<7. Exemplary Hardware Configuration>

A process of the action recording device 100 in accordance with thisembodiment can be executed either by hardware or software. In this case,the action recording device 100 can be configured as shown in FIG. 46.Hereinafter, an exemplary hardware configuration of the action recordingdevice 100 in accordance with this embodiment will be described withreference to FIG. 46.

The action recording device 100 in accordance with this embodiment canbe implemented by a processing device such as a computer as describedabove. As shown in FIG. 46, the action recording device 100 includes aCPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, RAM(Random Access Memory) 103, and a host bus 104 a. In addition, theaction recording device 100 also includes a bridge 104, an external bus104 b, an interface 105, an input device 106, an output device 107, astorage device (HDD) 108, a drive 109, a connection port 111, and acommunication device 113.

The CPU 101 functions as an arithmetic processing unit and a controlunit, and controls the entire operation within the action recordingdevice 100 in accordance with various programs. The CPU 101 may also bea microprocessor. The ROM 102 stores programs, operation parameters, andthe like used by the CPU 101. The RAM 103 temporarily stores programsused in the execution of the CPU 101, parameters that change asappropriate during the execution, and the like. These units are mutuallyconnected via the host bus 104 a including a CPU bus or the like.

The host bus 104 a is connected to the external bus 104 b such as a PCI(Peripheral Component Interconnect/Interface) bus via the bridge 104.Note that the host bus 104 a, the bridge 104, and the external bus 104 bneed not necessarily be arranged separately, and the functions of suchcomponents may be integrated into a single bus.

The input device 106 includes an input means for a user to inputinformation, such as a mouse, a keyboard, a touch panel, a button, amicrophone, a switch, or a lever; an input control circuit thatgenerates an input signal on the basis of a user input and outputs thesignal to the CPU 101; and the like. The output device 107 includes adisplay device such as, for example, a liquid crystal display (LCD)device, an OLED (Organic Light Emitting Diode) device, or a lamp; and anaudio output device such as a speaker.

The storage device 108 is a device for storing data, constructed as anexample of a storage unit of the action recording device 100. Thestorage device 108 can include a storage medium, a recording device thatrecords data on the storage medium, a reading device that reads datafrom the storage medium, a deletion device that deletes data recorded onthe storage medium, and the like. The storage device 108 includes, forexample, a HDD (Hard Disk Drive). The storage device 108 stores programsand various data for driving the hard disk and executed by the CPU 101.

The drive 109 is a reader/writer for a storage medium, and isincorporated in or externally attached to the action recording device100. The drive 109 reads information recorded on a removable storagemedium such as a magnetic disk, an optical disc, a magnetooptical disk,or semiconductor memory that is mounted, and outputs the information tothe RAM 103.

The connection port 111 is an interface for connection to an externaldevice, and is, for example, a connection port for connection to anexternal device that can transmit data via a USB (Universal Serial Bus).The communication device 113 is, for example, a communication interfaceincluding a communication device and the like for connection to thecommunication network 10. The communication device 113 may be any of acommunication device supporting a wireless LAN (Local Area Network), acommunication device supporting a wireless USB, or a wire communicationdevice that performs wire communication.

In the foregoing, a preferred embodiment of the present disclosure hasbeen described in detail with reference to the appended drawings, butthe technical scope of the present disclosure is not limited to theabove examples. A person skilled in the art may find various alterationsand modifications within the scope of the appended claims, and it shouldbe understood that they will naturally come under the technical scope ofthe present disclosure.

In the above embodiment, for example, the action representationgeneration unit 320 and the data management unit 330 are provided in theanalysis server 300 and the action representation generation unit 140 isprovided in the action recording device 100, but the present disclosureis not limited to such an example. For example, these functional unitsmay all be provided in the analysis server 300 or in the actionrecording device 100.

Additionally, the present technology may also be configured as below.

(1)

An information processing device including:

an action recognition unit that recognizes an operation action of a userbased on sensor information; and

an action representation generation unit that analyzes operation actiondata showing the operation action of the user recognized by the actionrecognition unit to generate an action segment represented by a meaningand content of the operation action from the operation action data.

(2)

The information processing device according to (1),

wherein dictionary data defining a relation of the higher meaning andcontent for the operation action is held, and

wherein the action representation generation unit estimates the meaningand content of the operation action from relations before and after theoperation action data arranged chronologically based on the dictionarydata to generate the action segment.

(3)

The information processing device according to (1) or (2),

wherein the action representation generation unit estimates the meaningand content of the operation action in accordance with a time period anda time of the operation action data to be analyzed to generate theaction segment.

(4)

The information processing device according to any one of (1) to (3),

wherein the action representation generation unit estimates the meaningand content of the operation action in accordance with positions of theoperation action data before and after the operation action data to beanalyzed to generate the action segment.

(5)

The information processing device according to any one of (1) to (4),

wherein hierarchical information showing a hierarchical relationshipabout the meaning and content is attached to the action segment.

(6)

The information processing device according to (5),

wherein the action representation generation unit displays the actionsegments based on a segmentation grain size deciding roughness ofsegmentation of the action segments and the hierarchical information.

(7)

The information processing device according to (6),

wherein the action representation generation unit combines or dividesthe action segments based on a size of the segmentation grain size anddisplays the combined or divided action segments.

(8)

The information processing device according to any one of (1) to (7),further including:

a typical action pattern generation unit that extracts one actionpattern from a plurality of action segment groups including the actionsegments of a predetermined unit based on a correlation between theaction segments.

(9)

The information processing device according to any one of (1) to (8),

wherein the action representation generation unit displays, in a displayunit, the action segments represented at least by a start time, an endtime, position information, and operation content of the operationaction by chronologically arranging the action segments.

(10)

The information processing device according to any one of (1) to (9),further including:

a feedback adjustment unit that corrects operation action estimationinformation that decides the operation action based on correctionfeedback from the user to the action segment generated by the actionrepresentation generation unit,

wherein the action representation generation unit generates the actionsegment constituting an action log from the operation action data basedon the operation action estimation data, and

wherein the feedback adjustment unit corrects the operation actionestimation information based on the correction feedback.

(11)

An information processing device including:

an action recognition unit that recognizes an operation action of a userbased on sensor information;

an action representation generation unit that generates an actionsegment constituting an action log from operation action data showingthe operation action of the user recognized by the action recognitionunit based on operation action estimation information that decides theoperation action; and

a feedback adjustment unit that corrects the operation action estimationinformation based on correction feedback from the user to the actionsegment generated by the action representation generation unit.

(12)

The information processing device according to (11),

wherein the operation action estimation information includes a pluralityof combinations of a probability distribution and a weighting factordepending on a location for a plurality of the operation actions, and

wherein the feedback adjustment unit corrects the probabilitydistribution or the weighting factor for each of the operation actionsbased on the correction feedback.

(13)

The information processing device according to (12),

wherein, when the correction feedback concerns action content, thefeedback adjustment unit corrects the probability distribution of theoperation action estimation information in accordance with content ofthe correction feedback.

(14)

The information processing device according to (11) or (12),

wherein, when the correction feedback concerns the location, thefeedback adjustment unit corrects the weighting factor of the operationaction estimation information in accordance with content of thecorrection feedback.

REFERENCE SIGN LIST

-   100 action recording device-   110 sensors-   120 action recognition unit-   122 sensor controller-   124 operation action recognition unit-   130 client interface unit-   140 action representation processing unit-   142 display processing unit-   144 input information processing unit-   150 display unit-   160 input unit-   200 action log server-   210 log server interface unit-   220 action log db-   300 analysis server-   310 analysis server interface unit-   320 action representation generation unit-   321 living action recognition unit-   322 hierarchical structure judgment unit-   330 data management unit-   331 data acquisition unit-   332 feedback adjustment unit-   333 analysis parameter db-   334 unit data storage db-   335 hierarchical information attached data storage db

1. An information processing device comprising: an action recognitionunit that recognizes an operation action of a user based on sensorinformation; and an action representation generation unit that analyzesoperation action data showing the operation action of the userrecognized by the action recognition unit to generate an action segmentrepresented by a meaning and content of the operation action from theoperation action data.
 2. The information processing device according toclaim 1, wherein dictionary data defining a relation of the highermeaning and content for the operation action is held, and wherein theaction representation generation unit estimates the meaning and contentof the operation action from relations before and after the operationaction data arranged chronologically based on the dictionary data togenerate the action segment.
 3. The information processing deviceaccording to claim 1, wherein the action representation generation unitestimates the meaning and content of the operation action in accordancewith a time period and a time of the operation action data to beanalyzed to generate the action segment.
 4. The information processingdevice according to claim 1, wherein the action representationgeneration unit estimates the meaning and content of the operationaction in accordance with positions of the operation action data beforeand after the operation action data to be analyzed to generate theaction segment.
 5. The information processing device according to claim1, wherein hierarchical information showing a hierarchical relationshipabout the meaning and content is attached to the action segment.
 6. Theinformation processing device according to claim 5, wherein the actionrepresentation generation unit displays the action segments based on asegmentation grain size deciding roughness of segmentation of the actionsegments and the hierarchical information.
 7. The information processingdevice according to claim 6, wherein the action representationgeneration unit combines or divides the action segments based on a sizeof the segmentation grain size and displays the combined or dividedaction segments.
 8. The information processing device according to claim1, further comprising: a typical action pattern generation unit thatextracts one action pattern from a plurality of action segment groupsincluding the action segments of a predetermined unit based on acorrelation between the action segments.
 9. The information processingdevice according to claim 1, wherein the action representationgeneration unit displays, in a display unit, the action segmentsrepresented at least by a start time, an end time, position information,and operation content of the operation action by chronologicallyarranging the action segments.
 10. The information processing deviceaccording to claim 1, further comprising: a feedback adjustment unitthat corrects operation action estimation information that decides theoperation action based on correction feedback from the user to theaction segment generated by the action representation generation unit,wherein the action representation generation unit generates the actionsegment constituting an action log from the operation action data basedon the operation action estimation data, and wherein the feedbackadjustment unit corrects the operation action estimation informationbased on the correction feedback.
 11. An information processing devicecomprising: an action recognition unit that recognizes an operationaction of a user based on sensor information; an action representationgeneration unit that generates an action segment constituting an actionlog from operation action data showing the operation action of the userrecognized by the action recognition unit based on operation actionestimation information that decides the operation action; and a feedbackadjustment unit that corrects the operation action estimationinformation based on correction feedback from the user to the actionsegment generated by the action representation generation unit.
 12. Theinformation processing device according to claim 11, wherein theoperation action estimation information includes a plurality ofcombinations of a probability distribution and a weighting factordepending on a location for a plurality of the operation actions, andwherein the feedback adjustment unit corrects the probabilitydistribution or the weighting factor for each of the operation actionsbased on the correction feedback.
 13. The information processing deviceaccording to claim 12, wherein, when the correction feedback concernsaction content, the feedback adjustment unit corrects the probabilitydistribution of the operation action estimation information inaccordance with content of the correction feedback.
 14. The informationprocessing device according to claim 12, wherein, when the correctionfeedback concerns the location, the feedback adjustment unit correctsthe weighting factor of the operation action estimation information inaccordance with content of the correction feedback.
 15. An informationprocessing method comprising: a step for recognizing an operation actionof a user based on sensor information; and a step for analyzingoperation action data showing the recognized operation action of theuser to generate an action segment represented by a meaning and contentof the operation action from the operation action data.
 16. Aninformation processing method comprising: a step for recognizing anoperation action of a user based on sensor information; a step forgenerating an action segment constituting an action log from operationaction data showing the recognized operation action of the user based onoperation action estimation information that decides the operationaction; and a step for correcting the operation action estimationinformation based on correction feedback from the user to the actionsegment.
 17. A computer program for causing a computer to function as aninformation processing device comprising: an action recognition unitthat recognizes an operation action of a user based on sensorinformation; and an action representation generation unit that analyzesoperation action data showing the operation action of the userrecognized by the action recognition unit to generate an action segmentrepresented by a meaning and content of the operation action from theoperation action data.
 18. A computer program for causing a computer tofunction as an information processing device comprising: an actionrecognition unit that recognizes an operation action of a user based onsensor information; an action representation generation unit thatgenerates an action segment constituting an action log from operationaction data showing the operation action of the user recognized by theaction recognition unit based on operation action estimation informationthat decides the operation action; and a feedback adjustment unit thatcorrects the operation action estimation information based on correctionfeedback from the user to the action segment generated by the actionrepresentation generation unit.